Apparatus and method of adaptive block filtering of target slice based on filter control information

ABSTRACT

Provided is an image processing apparatus including: area determination unit configured to determine whether or not an area of a control block functioning as control unit for filtering of an image includes a processing-target slice area of a plurality of slices formed in a frame of an encoded image; control information creation unit configured to create filter control information representing whether or not the filtering is performed for the area of the control block including a processing-target slice for each area of the control block including the processing-target slice when the area determination unit determines that the area of the control block includes the area of the processing-target slice; and filter unit configured to perform filtering for the image based on the filter control information created by the control information creation unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S.application Ser. No. 15/862,222, filed Jan. 4, 2018, which is acontinuation of U.S. application Ser. No. 14/934,548, filed Nov. 6,2015, now U.S. Pat. No. 9,955,161, which is a continuation applicationof U.S. application Ser. No. 14/032,766, filed Sep. 20, 2013, now U.S.Pat. No. 9,210,428, which is a continuation application of U.S.application Ser. No. 12/820,305, filed Jun. 22, 2010, now U.S. Pat. No.9,215,460, which contains subject matter related to that disclosed inJapanese Priority Patent Application JP 2009-179394 filed in the JapanPatent Office on Jul. 31, 2009. The entire contents of each of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing apparatus andmethod, and more particularly, to an image processing apparatus andmethod by which it is possible to suppress degradation of a encodingefficiency by locally controlling filtering during encoding or decoding.

2. Description of the Related Art

Recently, apparatuses complying with a standard such as the MPEG (MovingPicture Experts Group) are commercially available in both transmittinginformation from a broadcasting center or the like and receivinginformation in customer premises. Such apparatuses digitally handle andcompress image information using an orthogonal conversion such as adiscrete cosine transform and a motion compensation based oncharacteristic redundancy of the image information in order to transmitand accumulate information with an excellent efficiency.

Particularly, the MPEG2 (ISO(International Organization forStandardization)/IEC(International Electrotechnical Commission) 13818-2)defined as a general image encoding scheme is currently used in a widevariety fields such as both professional and consumer applications, bothinterlaced and the progressive scanning images, and is a standard whichencompasses both standard and high resolution images. According to theMPEG2 compression scheme, for example, a bit rate of 4 to 8 Mbps isallocated to a standard resolution interlaced image of 720×480 pixels,and a bit rate of 18 to 22 Mbps is allocated to a high resolutioninterlaced image of 1920×1088 pixels so that a high compression rate andan excellent image quality can be realized.

While the MPEG2 is mainly targeted to a high quality image encodingsuitable for a broadcasting, it fails to provide an encoding schemehaving a bit rate lower than that of the MPEG1, i.e., with a highercompression rate. As mobile phones are widely used, it is anticipatedthat demands on such an encoding scheme increases in the future.Accordingly, the MPEG4 encoding scheme has been standardized. As theimage encoding scheme, such a standard has been internationally approvedas ISO/IEC 14496-2 in December 1998.

Furthermore, recently, for the purpose of encoding a TV conferenceimage, H.26L (ITU-T (ITU Telecommunication Standardization Sector) Q6/16VCEG (Video encoding Experts Group)) is being standardized. According toH.26L, it is known that a large amount of computations are necessary toperform the encoding and the decoding in comparison with existingencoding schemes such as MPEG2 or MPEG4, but higher encoding efficiencycan be realized. In addition, currently, as one of the MPEG4 activities,a standardization for realizing higher encoding efficiency is beingdeveloped as a Joint Model of Enhanced-Compression Video encoding on thebasis of H.26L by including additional functionalities that are notsupported by H.26L. As a schedule of the standardization, aninternational standard based on the names of H.264 and MPEG4 Part 10(AVC (Advanced Video Coding)) has been approved in March 2003.

Recently, as a next-generation video encoding technology, an adaptiveloop filter (ALF) is being reviewed (e.g., refer to Yi-Jen Chiu and L.Xu, “Adaptive (Wiener) Filter for Video Compression,” ITU-T SG16Contribution, C437, Geneva, April 2008). This adaptive filter canalleviate block distortion or quantization distortion that is difficultto handle using a deblock filter by performing an optimal filtering foreach frame.

However, generally, since the image locally has various characteristics,the optimal filter coefficient is locally different. In the methoddisclosed in the Yi-Jen Chiu and L. Xu, “Adaptive (Wiener) Filter forVideo Compression,” ITU-T SG16 Contribution, C437, Geneva, April 2008,since the same filter coefficient is applied to all pixels in a singleframe, image quality may be improved in the entire frame but may bedegraded locally.

In this regard, a method of omitting the filtering in the area whereimage quality is locally degraded has been proposed (e.g., refer toTakeshi. Chujoh, et al., “Block-based Adaptive Loop Filter,” ITU-T SG16Q6 VCEG Contribution, AI18, Germany, July, 2008, and T. Chujoh, N. Wada,and G. Yasuda, “Quadtree-based Adaptive Loop Filter,” ITU-T SG16 Q6 VCEGContribution, VCEG-AK22(r1), Japan, April, 2009). In this method, theimage encoding apparatus controls whether or not the filtering isperformed for each control block by matching with a plurality of controlblocks compactly arranged on the area of the image. The image encodingapparatus establishes flag information for each block and performs theadaptive filtering based on the flag information. Similarly, the imagedecoding apparatus performs the adaptive filtering based on the flaginformation.

However, a method of performing the image encoding or decoding for eachslice by dividing a single frame into a plurality of slices(multi-slice) has been proposed. Takeshi. Chujoh, et al., “Block-basedAdaptive Loop Filter,” ITU-T SG16 Q6 VCEG Contribution, AI18, Germany,July, 2008, and T. Chujoh, N. Wada, and G. Yasuda, “Quadtree-basedAdaptive Loop Filter,” ITU-T SG16 Q6 VCEG Contribution, VCEG-AK22(r1),Japan, April, 2009 describe a method of establishing blocks for a singleentire frame and creating and transmitting flag information for all ofthe blocks but fail to describe how to process the flag information insuch a multi-slice case and how to create and use the flag information.

Takeshi. Chujoh, et al., “Block-based Adaptive Loop Filter,” ITU-T SG16Q6 VCEG Contribution, AI18, Germany, July, 2008, and T. Chujoh, N. Wada,and G. Yasuda, “Quadtree-based Adaptive Loop Filter,” ITU-T SG16 Q6 VCEGContribution, VCEG-AK22(r1), Japan, April, 2009 just describe that theimage encoding apparatus creates the flag information of all blockswithin the frame for a single slice within the frame. That is, even inthe case of the multi-slice, the image encoding apparatus is to createthe flag information of all blocks within the frame for each slice.

However, the flag information of the blocks of the area other than theprocessing-target slice is not necessary. The created flag informationis encoded together with the image data and included in the imagecompression information. That is, according to the methods disclosed inTakeshi. Chujoh, et al., “Block-based Adaptive Loop Filter,” ITU-T SG16Q6 VCEG Contribution, AI18, Germany, July, 2008, and T. Chujoh, N. Wada,and G. Yasuda, “Quadtree-based Adaptive Loop Filter,” ITU-T SG16 Q6 VCEGContribution, VCEG-AK22(r1), Japan, April, 2009, when the multi-slice isapplied, the image compression information may unnecessarily increaseand degrade encoding efficiency.

SUMMARY OF THE INVENTION

It is desirable to provide a method of suppressing degradation of theencoding efficiency by locally controlling the filtering during theencoding or the decoding.

According to an embodiment of the invention, there is provided an imageprocessing apparatus including: area determination means configured todetermine whether or not an area of a control block functioning ascontrol unit for filtering of an image includes a processing-targetslice area of a plurality of slices formed in a frame of an encodedimage; control information creation means configured to create filtercontrol information representing whether or not the filtering isperformed for the area of the control block including aprocessing-target slice, for each area of the control block includingthe processing-target slice when the area determination means determinesthat the area of the control block includes the area of theprocessing-target slice; and filter means configured to performfiltering for the image based on the filter control information createdby the control information creation means.

The image processing apparatus may further include slice area specifyingmeans configured to specify the processing-target slice area and controlblock area specifying means that specifies the area of the controlblock, wherein the area determination means may determine whether or notthe area of the control block specified by the control block areaspecifying means includes the processing-target slice area specified bythe slice area specifying means.

The image processing apparatus may further include control meansconfigured to determine whether or not the processing-target controlblock area includes the processing-target slice area and control thefilter means to perform filtering when a value of the filter controlinformation of the control block determined to include theprocessing-target slice area represents that the filtering is performed,wherein the filter means may perform the filtering for theprocessing-target control block area under control of the control means.

The image processing apparatus may further include encoding meansconfigured to create encoded data by encoding the image, wherein theencoding means may encode the filter control information created by thecontrol information creation means and add the encoded filter controlinformation to the encoded data.

The image processing apparatus may further include filter coefficientcalculation means configured to calculate a filter coefficient of thefiltering, wherein the encoding means encodes the filter coefficientcalculated by the filter coefficient calculation means and adds theencoded filter coefficient to the encoded data.

According to another embodiment of the invention, there is provided animage processing method including the steps of: in area determinationmeans of an image processing apparatus, determining whether or not anarea of a control block functioning as control unit of filtering for animage includes a processing-target slice area of a plurality of slicesformed in a frame of an encoded image; in control information creationmeans of the image processing apparatus, creating filter controlinformation representing whether or not the filtering is performed forthe area of the control block including the processing-target slice, foreach control block including the processing-target slice when it isdetermined that the area of the control block includes theprocessing-target slice area; and in filter means of the imageprocessing apparatus, performing filtering for the image based on thecreated filter control information.

According to still another embodiment of the invention, there isprovided an image processing apparatus including: control meansconfigured to, for each control block functioning as control unit offiltering for an image, determine whether or not the area of the controlblock of the image includes a processing-target slice area of aplurality of slices formed in a frame of the image and perform controlsuch that the filtering is performed for the area of the control blockof the image when a value of filter control information representingwhether or not the filtering is performed for the control blockdetermined to include the processing-target slice area represents thatthe filtering is performed; and filter means configured to perform thefiltering for the area of the control block of the image under controlof the control means.

The image processing apparatus may further include decoding meansconfigured to create the image by decoding encoded data obtained byencoding the image, wherein the decoding means decodes the encodedfilter control information added to the encoded data, the control meansperforms control such that the filtering is performed for the area ofthe control block of the image when the value of the filter controlinformation obtained by the decoding in the decoding means representsthat the filtering is performed, and the filter means performs thefiltering for the area of the control block of the image under controlof the control means.

According to still another embodiment of the invention, there isprovided an image processing method including the steps of: in controlmeans of the image processing apparatus, determining, for each controlblock functioning as control unit of filtering for an image, whether ornot the area of the control block of the image includes aprocessing-target slice area of a plurality of slices formed in a frameof the image and performing control such that the filtering is performedfor the area of the control block of the image when a value of filtercontrol information representing whether or not the filtering isperformed for the control block determined to include theprocessing-target slice area represents that the filtering is performed;and in filter means of the image processing apparatus, performing thefiltering for the area of the control block of the image under controlof the control means.

According to an embodiment of the invention, whether or not an area of acontrol block functioning as control unit of filtering for an imageincludes a processing-target slice area of a plurality of slices formedin a frame of an encoded image is determined. Filter control informationrepresenting whether or not the filtering is performed for the area ofthe control block including the processing-target slice is created foreach control block including the processing-target slice when it isdetermined that the area of the control block includes theprocessing-target slice area. The filtering is performed for the imagebased on the created filter control information.

According to another embodiment of the invention, for each control blockfunctioning as control unit of filtering for an image, whether or notthe area of the control block of the image includes a processing-targetslice area of a plurality of slices formed in a frame of the image isdetermined, and control is performed such that the filtering isperformed for the area of the control block of the image when a value offilter control information representing whether or not the filtering isperformed for the control block determined to include theprocessing-target slice area represents that the filtering is performed.The filtering is performed for the area of the control block of theimage under control of the control means.

According to embodiments of the invention, the image can be encoded ordecoded. Particularly, it is possible to suppress degradation of theencoding efficiency by locally controlling the filtering during theencoding or the decoding. For example, even when each frame of the imageis encoded or decoded by dividing into a plurality, it is possible tosuppress degradation of the encoding efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a mainconfiguration of an image encoding apparatus according to an embodimentof the invention.

FIG. 2 illustrates a motion prediction/compensation process of avariable block size.

FIG. 3 is a block diagram illustrating an example of a mainconfiguration of a control information creation unit.

FIGS. 4A, 4B, and 4C illustrate an ALF block and a filter block flag.

FIG. 5 illustrates an example of a multi-slice.

FIGS. 6A and 6B illustrate a processing for a slice 0.

FIGS. 7A and 7B illustrate a processing for a slice 1.

FIGS. 8A and 8B illustrate a processing for a slice 1 according to anembodiment of the invention.

FIG. 9 is a block diagram illustrating an example of a mainconfiguration of an adaptive filtering unit.

FIG. 10 is a flowchart illustrating an example of an encoding processflow.

FIG. 11 is a flowchart illustrating an example of a control informationcreation process flow.

FIG. 12 is a flowchart illustrating an example of a block informationcreation process flow.

FIG. 13 is a flowchart illustrating an example of an adaptive filteringprocess flow.

FIG. 14 is a block diagram illustrating an example of a mainconfiguration of an image decoding apparatus according to an embodimentof the invention.

FIG. 15 is a flowchart illustrating an example of a decoding processflow.

FIG. 16 is a flowchart illustrating another example of the blockinformation creation process flow.

FIGS. 17A to 17D illustrate another example of the ALF block and thefilter block flag.

FIG. 18 illustrates another example of the ALF block and the filterblock flag.

FIG. 19 illustrates a process at a multi-slice.

FIG. 20 is a block diagram illustrating an example of a mainconfiguration of a personal computer according to an embodiment of theinvention.

FIG. 21 is a block diagram illustrating an example of a mainconfiguration of a television set according to an embodiment of theinvention.

FIG. 22 is a block diagram illustrating an example of a mainconfiguration of a mobile phone device according to an embodiment of theinvention.

FIG. 23 is a block diagram illustrating an example of a mainconfiguration of a hard disc recorder according to an embodiment of theinvention.

FIG. 24 is a block diagram illustrating an example of a mainconfiguration of a camera according to an embodiment of the invention.

FIG. 25 illustrates an example of a macroblock.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention will now be described. The descriptionswill be made in the following sequence.

1. First Embodiment (Image Encoding Apparatus)

2. Second Embodiment (Image Decoding Apparatus)

3. Third Embodiment (Modification of a filter block flag creationprocess)

4. Fourth Embodiment (QALF)

5. Fifth Embodiment (Personal Computer)

6. Sixth Embodiment (Television Set)

7. Seventh Embodiment (Mobile Phone Device)

8. Eighth Embodiment (Hard Disc Recorder)

9. Ninth Embodiment (Camera)

1. First Embodiment

Device Configuration

FIG. 1 illustrates a configuration of the image encoding apparatus as animage processing apparatus according to an embodiment of the invention.

The image encoding apparatus 100 shown in FIG. 1 is an encoder forcompressively encoding an image based on, for example, H.264 and MPEG4,Part 10 Advanced Video Coding (hereinafter, denoted as H.264/AVC) andemploys an adaptive loop filter.

In the example of FIG. 1, the image encoding apparatus 100 includes anA/D (Analog/Digital) conversion unit 101, a picture sorting buffer 102,a computation unit 103, an orthogonal conversion unit 104, aquantization unit 105, a reversible encoder 106, and an accumulationbuffer 107. In addition, the image encoding apparatus 100 includes aninverse quantization unit 108, an inverse orthogonal conversion unit109, a computation unit 110, and a deblock filter 111. Furthermore, theimage encoding apparatus 100 includes a control information creationunit 112, an adaptive filtering unit 113, and a frame memory 114. Stillfurthermore, the image encoding apparatus 100 includes anintra-prediction unit 115, a motion compensation unit 116, a motionprediction unit 117, and a prediction image selection unit 118. Theimage encoding apparatus 100 also includes a rate control unit 119.

The A/D conversion unit 101 A/D converts the input image and outputs itto the picture sorting buffer 102 to store it. The picture sortingbuffer 102 sorts the images of frames stored in a display sequence intoan encoding frame sequence according to the GOP (Group of Pictures)structure.

The computation unit 103 subtracts a prediction image from theintra-prediction unit 115 selected by the prediction image selectionunit 118 or a prediction image from the motion compensation unit 116from the image read from the picture sorting buffer 102 and outputs thedifference information thereof to the orthogonal conversion unit 104.The orthogonal conversion unit 104 performs orthogonal conversion suchas a discrete cosine transform or a Karhunen and Loeve transformaccording to the difference information from computation unit 103 forthe difference information to output a conversion coefficient thereof.The quantization unit 105 quantizes the conversion coefficient outputfrom the orthogonal conversion unit 104.

The quantized conversion coefficient output from the quantization unit105 is input to the reversible encoder 106. The reversible encoder 106compresses the quantized conversion coefficient by performing areversible encoding such as a variable length encoding or an arithmeticencoding.

The reversible encoder 106 obtains information representing theintra-prediction or the like from the intra-prediction unit 114 andobtains information representing the inter-prediction mode from themotion prediction unit 117. Hereinafter, the information representingthe intra-prediction will be also referred to as intra-prediction modeinformation. In addition, the information representing the informationmode representing the inter-prediction will be hereinafter referred toas inter-prediction mode information.

The reversible encoder 106 obtains the control information of theadaptive filtering performed by the adaptive filtering unit 113 from thecontrol information creation unit 112.

The reversible encoder 106 encodes the control information of theadaptive filtering, the information representing the intra-prediction orthe inter-prediction mode, the quantization parameters, or the like aswell as the quantized conversion coefficient and sets (multiplexes) themas part of the header information in the compression image. Thereversible encoder 106 supplies the accumulation buffer 107 with theencoded data to accumulate them.

For example, the reversible encoder 106 performs reversible encoding ofa variable length encoding or an arithmetic encoding. As an example ofthe variable length encoding, a context-adaptive variable length coding(CAVLC) defined by H.264/AVC methods may be used. As an example of thearithmetic encoding, a context-adaptive binary arithmetic coding (CABAC)may be used.

The accumulation buffer 107 temporarily stores the data supplied fromthe reversible encoder 106 and outputs the data as compression imagesencoded by H.264/AVC methods, for example, to a recorder or atransmission line or the like located in a subsequent stage not shown inthe figures at a predetermined timing.

In addition, the conversion coefficient quantized by the quantizationunit 105 is also input to the inverse quantization unit 108. The inversequantization unit 108 inversely quantizes the quantized conversioncoefficient according to a method corresponding to the quantization ofthe quantization unit 105 and supplies the obtained conversioncoefficient to the inverse orthogonal conversion unit 109.

The inverse orthogonal conversion unit 109 performs inverse orthogonalconversion for the supplied conversion coefficient using a methodcorresponding to the orthogonal conversion processing by the orthogonalconversion unit 104. The output resulting from the inverse orthogonalconversion is supplied to the computation unit 110. The computation unit110 adds the prediction image supplied from prediction image selectionunit 118 to the inverse orthogonal conversion result supplied from theinverse orthogonal conversion unit 109, that is, the restored differenceinformation, to obtain a locally-decoded image (decoding image). Theaddition result is supplied to the deblock filter 111.

The deblock filter 111 removes block distortion from the decoding image.The deblock filter 111 supplies the distortion removal result to thecontrol information creation unit 112 and the adaptive filtering unit113.

The control information creation unit 112 obtains the decoding imagesupplied from the deblock filter 111 and the current input image readfrom the picture sorting buffer 102 and creates control information ofthe adaptive filter performed in the adaptive filtering unit 113therefrom. While details thereof will be described below, the controlinformation includes a filter coefficient, a block size, a filter blockflag, or the like.

The control information creation unit 112 supplied the created controlinformation to the adaptive filtering unit 113. In addition, the controlinformation creation unit 112 also supplies the created controlinformation to the reversible encoder 106. As described above, thecontrol information may be reversibly compressed by the reversibleencoder 106 and included (multiplexed) in the image compressioninformation. That is, the control information is sent to the imagedecoding apparatus together with the image compression information.

The adaptive filtering unit 113 performs filtering for the decodingimage supplied from the deblock filter 111 using the filter block flag,the block size specification, and the filter coefficient of the controlinformation supplied from the control information creation unit 112 andso on. For example, a Wiener filter may be used as such a filter.Needless to say, various filters other than the Wiener filter may beused. The adaptive filtering unit 113 supplies the filtering result tothe frame memory 114 and accumulates them as the reference image.

The frame memory 114 outputs the accumulated reference image to themotion compensation unit 116 and the motion prediction unit 117 at apredetermined timing.

In the image encoding apparatus 100, for example, an I-picture, aB-picture, and a P-picture from the picture sorting buffer 102 aresupplied to the intra-prediction unit 115 as an image for theintra-prediction (also, referred to as an intra-processing). Inaddition, the B-picture and the P-picture read from the picture sortingbuffer 102 are supplied to the motion prediction unit 117 as an imagefor the inter-prediction (also, referred to as an inter-processing).

The intra-prediction unit 115 performs the intra-prediction processingof all the candidates of the intra-prediction mode based on theintra-prediction image read from the picture sorting buffer 102 and thereference image supplied from the frame memory 114 to create theprediction image.

In the intra-prediction unit 115, the information on theintra-prediction mode applied to the corresponding block/macroblock istransmitted to the reversible encoder 106 and encoded as part of theheader information in the image compression information. According tothe H.264 image information encoding scheme, an intra 4×4 predictionmode, an intra 8×8 prediction mode, and an intra 16×16 prediction modeare defined for the luminance signal, and a prediction mode independentfrom the luminance signal may be defined for each macroblock for thecolor difference signal. In the intra 4×4 prediction mode, a singleintra-prediction mode is defined for each 4×4 luminance block. In theintra 8×8 prediction mode, a single intra-prediction mode is defined foreach 8×8 luminance block. In the intra 16×16 prediction mode and thecolor difference signal, a single prediction mode is defined for eachsingle macroblock.

The intra-prediction unit 115 calculates a cost function value for theintra-prediction mode that creates the prediction image, and selects theintra-prediction mode that gives a minimum value of the calculated costfunction values as an optimal intra-prediction mode. Theintra-prediction unit 115 supplies the prediction image generated by theoptimal intra-prediction mode to the prediction image selection unit118.

The motion prediction unit 117 obtains the image information (inputimage) supplied from the picture sorting buffer 102 and the imageinformation (decoding image) corresponding to the reference framesupplied from the frame memory 114 for the image to be inter-coded andcalculates the motion vector. The motion prediction unit 117 suppliesthe motion vector information representing the calculated motion vectorto the reversible encoder 106. The motion vector information islosslessly compressed by the reversible encoder 106 and included in theimage compression information. That is, the motion vector information issent to the image decoding apparatus together with the image compressioninformation.

In addition, the motion prediction unit 117 also supplies the motionvector information to the motion compensation unit 116.

The motion compensation unit 116 performs the motion compensation inresponse to the motion vector information supplied from the motionprediction unit 117 to create the inter-prediction image information.The motion compensation unit 116 supplies the created prediction imageinformation to the prediction image selection unit 118.

The prediction image selection unit 118 supplies the output of theintra-prediction unit 115 to the computation unit 103 in the case of theintra-coding and supplies the output of the motion compensation unit 116to the computation unit 103 in the case of the inter-coding.

The rate control unit 119 controls the rate of the quantization motionin the quantization unit 105 so as not to generate overflow or underflowbased on the compressed images accumulated in the accumulation buffer107.

According to the MPEG (Moving Picture Experts Group) 2 scheme, units ofthe motion prediction/compensation are a motion compensation block, andeach of the motion compensation blocks may include independent motionvector information. The motion compensation block size may include 16×16pixels in the case of the frame motion compensation mode or 16×8 pixelsfor each of the first and second fields in the case of the field motioncompensation mode.

On the contrary, according to the AVC (Advanced Video coding) scheme, asshown in the upper half of FIG. 2, a single macroblock including 16×16pixels is divided into several partitions of 16×16, 16×8, 8×16, or 8×8pixels, each of which may have independent motion vector information.Furthermore, as shown in the lower half of FIG. 2, the 8×8 partition maybe divided into several sub-partitions of 8×8, 8×4, 4×8, or 4×4 pixels,each of which may have independent motion vector information. The motionprediction/compensation is performed by using this motion compensationblock as unit.

FIG. 3 is a block diagram illustrating an example of a mainconfiguration of the control information creation unit 112.

The control information creation unit 112 creates control informationused in the adaptive filter (ALF: Adaptive Loop Filter), which is a loopfilter, performed in the adaptive filtering unit 113 as described above.The control information creation unit 112 creates, for example, a filtercoefficient, an ALF block size, and a filter block flag as the controlinformation.

The control information creation unit 112 has a filter coefficientcalculation unit 131 and a block information creation unit 132.

The filter coefficient calculation unit 131 obtains the decoding imagesupplied from the deblock filter 111 and the current input image readfrom the picture sorting buffer 102 and calculates the filtercoefficient of the ALF for each frame based on them.

The block information creation unit 132 determines the ALF block sizebased on the decoding image supplied from the deblock filter 111 and thefilter coefficient calculated by the filter coefficient calculation unit131 and creates the filter block flag for each ALF block within theprocessing-target slice.

Here, the ALF block and the filter block flag will be described. FIGS.4A to 4C illustrate the ALF block and the filter block flag.

As described above, in the adaptive filter, the filter coefficient isset for each frame. That is, the optimal filtering is performed on aframe-by-frame basis. However, generally, the frame image is not uniformin its entirety and locally has various characteristics. For thisreason, the optimal filter coefficient is locally different. Therefore,in the filtering using the filter coefficient determined for each frameas described above, the image quality may be improved in the entireframe but may be locally degraded on the contrary.

In this regard, a block-based adaptive loop filter (BALF) was conceivedin which the filtering is not performed for the area where the imagequality is locally degraded.

The frame 151 of FIG. 4A illustrates a decoding image after the deblockfiltering. As shown in FIG. 4B, the block information creation unit 132compactly arranges a plurality of ALF blocks 152, that are controlblocks as a control unit of the adaptive filtering each of which islocally performed, on the entire area of the frame 151. While the areawhere the ALF blocks 152 are arranged may not be equal to the area ofthe frame 151, it includes at least the entire area of the frame. As aresult, the area of the frame 151 is divided into the areas of each ALFblock 152.

The block information creation unit 132 determines the vertical size(the bidirectional arrow 154) and the horizontal size (the bidirectionalarrow 153) of the ALF block 152. The ALF block size may be designatedfor each slice using any one of, for example, 8×8, 16×16, 24×24, 32×32,48×48, 64×64, 96×96, or 128×128 pixels. In addition, information fordesignating that ALF block size is referred to as a block size index.

If the block size is determined, the frame size is fixed. Thus, thenumber of ALF blocks per single frame is also determined.

As shown in FIG. 4C, the block information creation unit 132 establishesthe filter block flag 155 for controlling whether or not the filteringis performed for each ALF block 152. For example, a filter block flag155 is set to a value of “1” for the area where the image quality isimproved by the adaptive filter, and the filter block flag 155 is set toa value of “0” for the area where the image quality is degraded by theadaptive filter. In the filter block flag 155, a value of “1” representsthat the filtering is performed, and a value of “0” represents that thefilter is not performed.

The adaptive filtering unit 113 controls the adaptive filtering based onthe value of the filter block flag 155. For example, the adaptivefiltering unit 113 performs the filtering for the area of the ALF block152 where the filter block flag 155 is set to a value of “1”, and doesnot perform the filtering for the area of the ALF block 152 where thefilter block flag 155 is set to a value of “0”.

In addition, the aforementioned block size index and the filter blockflag may include the slice header of the image compression informationand may be sent from the image encoding apparatus 100 to the imagedecoding apparatus. One or more filter block flags corresponding to thenumber of the ALF blocks may be included in the slice header, forexample, in the order of a raster scanning.

Therefore, as the ALF block size decreases, the more accurate filtercontrol can be made, so that the ALF filter may be operated moreappropriately. However, as the ALF block size decreases, the bit amountof the filter block flag increases. That is, as the ALF block sizedecreases, the encoding efficiency of the image compression informationdecreases. As a result, there is a tradeoff relationship betweenperformance of the adaptive filter and encoding efficiency of the imagecompression information.

The number of the ALF blocks can be calculated according to thefollowing equation (1).N _(ALFBLOCK)=floor((16×N _(MBw) +N _(SIZE)−1)/N _(SIZE))×floor((16×N_(MBh) +N _(SIZE)−1)/N _(SIZE))  [Equation 1]

In the equation (1), NALFBLOCK denotes the number of the ALF blocks. Inaddition, NMBw denotes the number of macroblocks in the horizontaldirection of the picture, and NMBh denotes the number of macroblocks inthe vertical direction of the picture. Furthermore, NSIZE denotes thesize of one side of the ALF block. In addition, floor[x] denotes afunction for truncating digits under a decimal point to obtain aninteger number.

However, according to H.264/AVC, it is possible to divide a single frameinto a plurality of slices and output the image compression informationto each slice. FIG. 5 illustrates an example of a multi-slice. In theexample of FIG. 5, the frame 151 is divided into three slices includinga slice 0, a slice 1, and a slice 2.

The image encoding apparatus creates and outputs the image compressioninformation with a shorter interval by outputting the image compressioninformation in a more detailed slice unit than that of such a frame.That is, it is possible to allow the image decoding apparatus whichdecodes that image compression information to initiate decoding of theimage compression information at an earlier time. In other words, it ispossible to shorten the delay time after the image is input until theimage is output through the encoding and decoding.

The reference document Takeshi. Chujoh, et al., “Block-based AdaptiveLoop Filter” ITU-T SG16 Q6 VCEG Contribution, AI18, Germany, July, 2008’which describes the BALF does not disclose this multi-slice. That is, itdoes not describe that the ALF block is set for the entire frame. Inorder to create the filter block flag, it is necessary to provide thedeblock-filtered decoding image. Therefore, if the filter block flag forthe entire frame is created in a single time, it is necessary to standby until the image is encoded for all the slices within the frame (untilthe deblock-filtered decoding image is obtained). In this case, since adelay time increases accordingly, there is no meaning to process theimage on a slice-by-slice basis.

Therefore, in the case of the multi-slice, it is preferred to establishthe ALF block for each slice and create the filter block flag in orderto suppress the delay time from increasing. However, as described above,the ALF block is established for the entire frame. That is, the ALFblock is established for the entire frame as described above. That is,the ALF block is established for the entire frame in each slice, and itis a concern that an unnecessary ALF block may be established in theareas other than the slice area.

For example, in FIG. 5, when the slice 0 is processed as shown in FIG.6A, the ALF block 152 for the entire frame 151 is established for thearea of the slice 0 represented by the border line 161 as shown in FIG.6B.

Similarly, for example, in the example of FIG. 5, when the slice 1 isprocessed as shown in FIG. 7A, the ALF block 152 for the entire frame151 is established for the area of the slice 1 represented by the borderline 162 as shown in FIG. 7B.

The ALF block 152 hatched in FIGS. 6B and 7B is a block out of the areaof the slice 0 or 1 and also a block unnecessary to process the area ofthe slice 0 or 1.

As described above, the filter block flag 155 is established in each ALFblock. That is, in the case of the multi-slice, an unnecessary filterblock flag is created, and the data amount of the image compressioninformation increases, so that the encoding efficiency may be degraded.

In this regard, the block information creation unit 132 of the controlinformation creation unit 112 of FIG. 3 creates the filter block flagand the ALF block including the processing-target slice area in order tosuppress degradation of the encoding efficiency.

For example, in the example of FIG. 5, when the slice 1 is processed asshown in FIG. 8A, the block information creation unit 132 establishesthe ALF block 152 including the area of the slice 1 represented by theborder line 162 as shown in FIG. 8B and creates the filter block flagonly for this ALF block 152.

As a result, only the filter block flag demanded by each slice is addedto the image compression information. Therefore, in comparison with thecase where the filter block flag of the entire frame is created for eachslice as described above, the bit amount of the filter block flag isreduced, and degradation of the encoding efficiency is suppressed. Inaddition, since the image compression information including the controlinformation is created for each slice, the image decoding apparatus caninitiate the image decoding process on a slice-by-slice basis.Therefore, in comparison with the case where the filter block flag iscreated for each frame, it is possible to suppress the delay time fromincreasing.

Returning to FIG. 3, the block information creation unit 132 includes aprocessing-target slice area specifying unit 141, an ALF block settingunit 142, a processing-target ALF block area specifying unit 143, adetermination unit 144, and a filter block flag creation unit 145.

The processing-target slice area specifying unit 141 specifies thelocation of the processing-target slice area supplied as the decodingimage in the entire frame.

The ALF block setting unit 142 determines the ALF block size and setsthe ALF block 152 for the entire frame. Since the size of the entireframe area is previously determined, the ALF block setting unit 142 canspecify the number of the ALF blocks of the entire frame based on thedetermined block size.

The processing-target ALF block area specifying unit 143 selects theprocessing-target ALF block one by one from the ALF block 152 set by theALF block setting unit 142 and specifies the location of the selectedprocessing-target ALF block area.

The determination unit 144 determines whether or not theprocessing-target ALF block area includes the processing-target slicearea. The filter block flag creation unit 145 creates the filter blockflag of the ALF block for which the determination unit 144 determinesthat the processing-target slice area is included. The filter block flagcreation unit 145 performs the adaptive filtering for theprocessing-target ALF block area using the filter coefficient calculatedby the filter coefficient calculation unit 131 and determines the valueof filter block flag based on whether or not the image quality has beenimproved after the filtering.

The filter block flag creation unit 145 outputs the control informationsuch as the filter block flag or the ALF block size.

FIG. 9 is a block diagram illustrating an example of a mainconfiguration of the adaptive filtering unit 113 of FIG. 1.

The adaptive filtering unit 113 performs filtering for the decodingimage supplied from the deblock filter 111 using the control informationsupplied from the control information creation unit 112.

As shown in FIG. 9, the adaptive filtering unit 113 includes acontroller 171, an adaptive filter 172, and a selector 173.

The controller 171 performs control for the adaptive filter 172 and theselector 173. For example, the controller 171 obtains controlinformation from the control information creation unit 112. In addition,the controller 171 supplies the filter coefficient included in theobtained control information to the adaptive filter 172 and sets it.Furthermore, the controller 171 specifies the location of the ALF blockarea to be processed based on the ALF block size included in the controlinformation. In addition, the controller 171 performs control for theadaptive filter 172 based on the value of the filter block flag includedin the control information to perform filtering for each ALF block areaif necessary and, at the same time, performs control for the operationof the selector 173.

The adaptive filter 172 performs filtering for the area specified by thecontroller 171 as the processing target ALF block in the decoding imagesupplied from the deblock filter 111 using the filter coefficient set bythe controller 171. The adaptive filter 172 supplies the filteringresult to the selector 173.

The selector 173 selects any one of the decoding image (the decodingimage not subjected to the adaptive filtering) controlled by thecontroller 171 and supplied from the deblock filter 111 and the decodingimage (the decoding image subjected to the adaptive filtering) suppliedfrom the adaptive filter 172, supplies it to the frame memory 114, andaccumulates it as a reference image.

That is, the adaptive filtering unit 113 performs filtering only for thearea where the filtering is performed by the filter block flag (the areadetermined that it is possible to improve the image quality by thefiltering) of the decoding image supplied from the deblock filter 111.

Process Flow

Next, a process flow using each unit described above will be described.Initially, the process flow of the encoding performed by the imageencoding apparatus 100 will be described with reference to the flowchartof FIG. 10.

In step S101, the A/D conversion unit 101 performs A/D conversion forthe input image. In step S102, the picture sorting buffer 102 stores theA/D converted image and performs sorting from the sequence of displayingeach picture to the encoding sequence.

In step S103, the computation unit 103 computes a difference between thepredicted image and the sorted image in the process of step S102. Thepredicted image is supplied to the computation unit 103 through theprediction image selection unit 118 from the motion compensation unit116 in the case of the inter-prediction or from the intra-predictionunit 115 in the case of the intra-prediction.

The difference data has a smaller data amount in comparison with theoriginal image data. Therefore, it is possible to reduce the data amountin comparison with the case where the image is encoded without change.

In step S104, the orthogonal conversion unit 104 performs orthogonalconversion for the difference information created by step S103.Specifically, the conversion coefficient is output by the orthogonalconversion such as a discrete cosine transform or Karhunen and Loevetransform. In step S105, the quantization unit 105 quantizes theconversion coefficient. In this quantization, the rate is controlled asdescribed in the process of step S119 described below.

The quantized difference information as described above is locallydecoded as described below. That is, in step S106, the inversequantization unit 108 performs inverse quantization for the conversioncoefficient quantized by the quantization unit 105 using thecharacteristic corresponding to the characteristic of the quantizationunit 105. In step S107, the inverse orthogonal conversion unit 109performs inverse orthogonal conversion for the conversion coefficientinversely-quantized by the inverse quantization unit 108 using thecharacteristic corresponding to the characteristic of the orthogonalconversion unit 104.

In step S108, the computation unit 110 adds the predicted image inputthrough the prediction image selection unit 118 to the locally-decodeddifference information to create the locally-decoded image (the imagecorresponding to the input to the computation unit 103). In step S109,the deblock filter 111 performs filtering for the image output by thecomputation unit 110. As a result, the block distortion is removed.

As the aforementioned process is performed for a single slice, thecontrol information creation unit 112 creates control information usedin the adaptive filtering in step S110. Details of the controlinformation creation process will be described below in detail.

As control information such as the ALF block size and the filter blockflag is created through the process of step S110, the adaptive filteringunit 113 performs adaptive filtering for the deblock-filtered decodingimage in step S109 using the control information in step S111. Detailsof this adaptive filtering will be described below.

In step S112, the frame memory 114 stores the image subjected to theadaptive filtering in step S111.

In step S113, the intra-prediction unit 115 performs theintra-prediction process of the intra-prediction mode. In step S114, themotion prediction unit 117, and the motion compensation unit 116performs the inter-motion prediction/compensation processing of theinter-prediction mode.

In step S115, the prediction image selection unit 118 selects any one ofthe prediction image created by the intra-prediction processing or theprediction image created by the inter-motion prediction/compensationprocessing depending on the prediction mode of the processing-targetframe. The prediction image selection unit 118 supplies the selectedprediction image to the computation unit 103 and the computation unit110. This prediction image is used in operations of steps S103 and S108as described above.

In step S116, the reversible encoder 106 encodes the quantizedconversion coefficient output from the quantization unit 105. That is,the difference image is compressed using a reversible encoding such as avariable length encoding encoding or an arithmetic encoding. In thiscase, the reversible encoder 106 also encodes the control informationcreated in step S110, the intra-prediction mode information of theintra-prediction process in step S113, the inter-prediction mode of theinter-motion prediction/compensation process of step S114, or the like.

In step S117, the reversible encoder 106 embeds (describes) the metadatasuch as the encoded control information into a slice header. Thismetadata is read and used when the image is decoded. By including(multiplexing) the metadata necessary to perform the decoding process inthe slice header as described above, it is possible to perform thedecoding process using a more accurate unit than a frame unit andsuppress the delay time from increasing.

In step S118, the accumulation buffer 107 accumulates the differenceimage as the compression image. The compression image accumulated in theaccumulation buffer 107 is appropriately read and transmitted to thedecoding side through a transmission line.

In step S119, the rate control unit 119 performs control for the rate ofthe quantization operation of the quantization unit 105 so as not togenerate overflow or underflow based on the compression imageaccumulated in the accumulation buffer 107.

Next, in step S110 of FIG. 10, an example of the control informationcreation process flow performed by the control information creation unit112 will be described with reference to the flowchart of FIG. 11.

As the control information creation process is initiated, the filtercoefficient calculation unit 131 of the control information creationunit 112 calculates the filter coefficient using the input imagesupplied from the picture sorting buffer 102 and the deblock-filtereddecoding image supplied from the deblock filter 111 in step S131. Forexample, the filter coefficient calculation unit 131 determines a valueof the filter coefficient such that a residual difference between theinput image and the decoding image becomes minimum.

As the filter coefficient is calculated, the block information creationunit 132 performs creation of the block information including the ALFblock size or the filter block flag in step S132. Details of the blockinformation creation process will be described below. As the blockinformation is created, the process returns to step S110 of FIG. 10, andthe process subsequent to step S111 is executed.

In addition, the filter coefficient calculated in step S131 may beobtained on a frame-by-frame basis. In this case, the process of stepS131 may be performed only for a predetermined slice within a frame(such as a slice having a predetermined value of an identificationnumber, for example, 0 within the frame or a slice initially processedwithin the frame or the like), and the value may be similarly used inother slices. In addition, any image can be used to calculate the filtercoefficient. For example, the filter coefficient may be calculated basedon a past frame image.

Next, an example of the block information creation process flow executedin step S132 of FIG. 11 will be described with reference to theflowchart of FIG. 12.

As the block information creation process is initiated, theprocessing-target slice area specifying unit 141 specifies theprocessing-target slice area in step S151.

The corresponding processing-target slice area may be recognized byidentifying a macroblock included in the corresponding slice andidentifying a pixel included in that macroblock. The processing-targetslice area specifying unit 141 obtains the leading end macroblockaddress of the corresponding slice from the slice header.

Here, the leading end macroblock address refers to a number attached tothe macroblock starting from the upper left of the image in the order ofa raster scanning. As shown in FIG. 5, the address of the upper leftmacroblock within the image (the frame 151) is set to 0. Since the slice0 is initiated from the upper left of the frame 151, the macroblockaddress of the leading end macroblock 156-1 of the slice 0 is 0. In thisorder, the macroblock address of the final macroblock 156-2 of the slice0 is set to E0. In addition, similar to this slice 0, the macroblockaddress of the leading end macroblock 157-1 of the slice 1 is set to S1,and the macroblock address of the final macroblock 157-2 is set to E1.Furthermore, the macroblock address of the leading end macroblock 158-1of the slice 2 is set to S2, and the macroblock address of the finalmacroblock 158-2 is set to E2.

As the corresponding slice is decoded, the macroblock address isincremented by 1 whenever the decoding process is completed for a singlemacroblock, and finally, to the final macroblock of the correspondingslice. In the final macroblock of the slice, a flag is set. As a result,it is possible to recognize all of the macroblock addresses of thecorresponding slice, i.e., from the leading end macroblock address tothe final macroblock address.

However, the image size of a single frame is represented by the numberof macroblocks in the SPS (sequence parameter set) of the AVC stream(image compression information). The parameterpic_height_in_map_units_minus1 denotes the number of macroblocks in avertical direction of the image. The parameter pic_width_in_mbs_minus1denotes the number of macroblocks in a horizontal direction of theimage.

Therefore, the location of the macroblock can be represented by thefollowing equations (2) and (3) from the macroblock address.mbx=macro block address 0012C694ic_width_in_mbs_minus1   (2)mby=floor[macro block address/pic_width_in_mbs_minus1]   (3)

In the equations (2) and (3), the parameter mbx represents where themacroblock is located from the leftmost side, and the parameter mbyrepresents where the macroblock is located from the uppermost side. Inaddition, the operator floor[z] represents an operation of truncatingdigits under the decimal point to make an integer number. The parameterA represents residual numbers obtained by dividing the number A by thenumber B.

Assuming that the macroblock size is determined as 16×16 pixels, thelocation in the vertical and horizontal directions of the upper leftpixel of the macroblock is set to (16×mbx, 16×mby). That macroblockincludes 16 pixels in the lower direction and 16 pixels in the rightwarddirection with respect to the location of the upper left pixel. As aresult, all pixels of the corresponding slice can be recognized. Thatis, the processing-target slice area is specified.

In step S152 of FIG. 12, the ALF block setting unit 142 determines theALF block size. In step S153, the ALF block setting unit 142 determinesthe number of the ALF blocks within the frame. Since the frame imagesize is previously determined, as the ALF block size is determined, itis possible to calculate the number of the ALF blocks (the number of ALFblocks within the frame) necessary to arrange the ALF blocks on theentire area using the upper left corner of the frame as an origin. Sincethe setup values of the vertical and horizontal sizes (the number ofpixels) of the ALF block are previously provided, the ALF block settingunit 142 determines the ALF block size and the number of ALF blocksbased on the setup values and arranges the ALF blocks in the decodingimage.

In addition, the number of the ALF blocks is calculated based on thefollowing equations (4) and (5).num_alf_block_x=floor[(16×(pic_width_in_mbs_minus1+1)+(alf_block_size−1))/alf_block_size]  (4)num_alf_block_y=floor[(16×(pic_height_in_map_units_minus1+1)+(alf_block_size−1))/alf_block_size]  (5)

In the equations (4) and (5), the parameters num_alf_block_x andnum_alf_block_y represent the horizontal and vertical numbers,respectively, of the ALF block included in the image. The parameteralf_block_size represents the size of a single side of the ALF block.Here, for the purpose of simple descriptions, it is assumed that the ALFblock is cubic. Needless to say, the horizontal and vertical sizes ofthe ALF block may be different from each other.

In step S154, the processing-target ALF block area specifying unit 143determines the processing-target ALF block. In step S155, theprocessing-target ALF block area specifying unit 143 specifies theprocessing-target ALF block area.

The location of the ith ALF block is represented in the followingequations (6) and (7).alf_block_x=i(num_alf_block_x−1)  (6)alf_block_y=floor[i/(num_alf_block_x−1)]  (7)

In the equations (6) and (7), the parameters alf_block_x and alf_block_yrepresent where the ith ALF block is located in the horizontal andvertical directions, respectively. The location of the upper left pixelof the ith ALF block can be obtained by multiplying each value of theparameters alf_block_x and aof_block_y by the parameter alf_block_size.That is, the location of the horizontal direction is set to16×alf_block_x, and the location of the vertical direction is set to16×alf_block_y. Therefore, the ith ALF block area occupies a range ofalf_block_size×alf_block_size with respect to the upper left pixel.

In step S156, the determination unit 144 determines whether or not theprocessing-target slice area is included in the processing-target ALFblock specified as described above.

If it is determined that the processing-target slice area is included inthe processing-target ALF block area, the process advances to step S157.In step S157, the filter block flag creation unit 145 creates the filterblock flag for that ALF block because the processing-target ALF block isthe ALF block demanded by the processing-target slice. In step S158, thefilter block flag creation unit 145 outputs the created filter blockflag.

If the process of step S158 is terminated, the process advances to stepS159. In addition, in step S156, if it is determined that theprocessing-target slice area is not included in the processing-targetALF block area, that ALF block is not necessary for theprocessing-target slice, and the process advances to step S159.

In step S159, the processing-target ALF block area specifying unit 143determines whether or not all of the ALF blocks within the frame havebeen processed. If it is determined that all of the ALF blocks are notprocessed, the process returns to step S154, and the subsequent processis iterated by setting the new ALF block as the processing target. Theprocessing-target ALF block area specifying unit 143 selects an ALFblock from the ALF block group arranged in the entire frame areaone-by-one as the processing-target ALF block in the order of the rasterscanning starting from the upper left ALF block when this loopprocessing is iterated.

In addition, in step S159, if it is determined that all of the ALFblocks within the frame are processed, the block information creationprocess is terminated, and the process returns to step S132 of FIG. 11.As the control information creation process is terminated, the processreturns to step S110 of FIG. 10, and the process subsequent to step S111is executed.

In addition, while, in the aforementioned descriptions, the upper leftpoint of the frame is set to an origin when the ALF blocks are arrangedin the entire frame image area, the location of this origin may bearbitrarily set. For example, the location of the origin may be set tothe lower left, the lower right, the upper right, or the center point.However, the location of the origin and how to arrange the ALF blocksare necessary to be previously set so as to be common between theencoding and the decoding.

While in the aforementioned descriptions, the order of selecting theprocessing-target ALF blocks is set to the raster scanning orderstarting from the upper left point, the selection order and the startinglocation may be arbitrarily set.

Next, an example of the adaptive filter process flow executed in stepS111 of FIG. 10 will be described with reference to the flowchart ofFIG. 13.

As the adaptive filtering is initiated, the decoding image of theprocessing-target slice is supplied to the adaptive filter 172 and theselector 173. In step S171, the controller 171 specifies thatprocessing-target slice area. Similar to the process of step S151 ofFIG. 12, the controller 171 obtains the leading end macroblock addressof the corresponding slice of the slice header, further detects the flagrepresenting the final macroblock, and specifies the area from theleading end macroblock address to the final macroblock address as theprocessing target slice area.

In step S172, the controller 171 obtains the filter coefficient createdby the control information creation unit 112 and sets that filtercoefficient in the adaptive filter 172. In step S173, the controller 171obtains the ALF block size determined by the control informationcreation unit 112 and sets (arranges) the ALF blocks having that ALFblock size in the entire frame area.

In step S174, similar to step S154 of FIG. 12, the controller 171determines one of the unprocessed ALF blocks of the ALF block groupestablished as described above as the processing-target ALF block. ThisALF block selection order is previously determined and common to theselection order of the control information creation unit 112.

In step S175, similar to step S155 of FIG. 12, the controller 171specifies the determined processing-target ALF block area.

In step S176, similar to step S156 of FIG. 12, the controller 171determines whether or not the processing-target slice area is includedin the processing-target ALF block area. If it is determined that theprocessing-target slice area is included, the process advances to stepS177.

In step S177, the controller 171 obtains the filter block flag of theprocessing-target ALF block created in the control information creationunit 112. Since the control information creation unit 112 creates thefilter block flag as described above, the filter block flag is suppliedonly to the ALF block including the processing-target slice area inpractice. Since the processing order of the ALF block is common to thecontrol information creation unit 112, the filter block flag is suppliedin the processing order of the ALF block. Therefore, the controller 171can obtain (employ) the filter block flag of the processing-target ALFblock by obtaining (employing) the filter block flag in that supplyorder.

In addition, the timing of supplying the filter block flag and thetiming of obtaining the filter block flag using the controller 171 maynot be the same. That is, the controller 171 may temporarily store thefilter block flag supplied from the control information creation unit112, for example, in an internal buffer or the like and read the filterblock flag from that buffer in the process of step S177. Even in thatcase, the controller 171 can obtain the filter block flag of theprocessing-target ALF block just by making the order of reading thefilter block flag to be the same as the order of supplying the filterblock flag from the control information creation unit 112, i.e., theaccumulation order in the buffer.

In step S178, the controller 171 determines whether or not the value ofthe filter block flag is 1. If the value of the filter block flag is 1,and it is instructed that the filtering is performed for theprocessing-target ALF block area, the process advances to step S179. Instep S179, the adaptive filter 172 is controlled by the controller 171,and the filtering is performed for the processing-target ALF block. Ifthe process of step S179 is terminated, the process advances to stepS180. In this case, the selector 173 is controlled by the controller 171in step S180, and the output of the adaptive filter 172 is selected andoutput to the frame memory 114. In other words, (a part of the area of)the decoding image subjected to the filtering is accumulated in theframe memory 114. As the process of step S180 is terminated, the processadvances to step S181.

In addition, in step S178, if the value of the filter block flag is 0,and it is instructed that the filtering is not performed for theprocessing-target ALF block area, the process of step S179 is omitted,and the process advances to step S180. In this case, in step S180, theselector 173 is controlled by the controller 171, and the output of thedeblock filter 111 is selected and output to the frame memory 114. Thatis, (a part of the area of) the decoding image not subjected to thefiltering is accumulated in the frame memory 114. As the process of stepS180 is terminated, the process advances to step S181.

In addition, in step S176, if it is determined that theprocessing-target slice area is not included in the processing-targetALF block area, the processing-target ALF block is an ALF block havingno relationship with the processing-target slice. Therefore, the processof steps S177 to S180 is omitted, and the process advances to step S181.

In step S181, the controller 171 determines whether or not all of theALF blocks within the frame are processed. If it is determined thatthere is any unprocessed ALF block, the process returns to step S174,and the subsequent process is iterated for a new processing-target ALFblock. The controller 171 selects an ALF block from the ALF block grouparranged in the entire frame area one-by-one as the processing-targetALF block in the order of the raster scanning starting from the upperleft ALF block when this loop processing is iterated.

In addition, in step S181, if it is determined that all of the ALFblocks within the frame are processed, the adaptive filtering isterminated. The process returns to step S111 of FIG. 10, and the processsubsequent to step S112 is iterated.

By performing the adaptive filtering as described above, the adaptivefiltering unit 113 can appropriately perform the filtering for theprocessing-target slice, demanded by the processing-target slice of aplurality of slices formed in the frame, based on the filter block flagof a part of the ALF block within the frame. As a result, the adaptivefiltering unit 113 can reduce block distortion or quantizationdistortion that is not handled by the deblock filter of theprocessing-target slice.

How to arrange the ALF blocks is previously determined. Therefore, inthe state that the ALF blocks are arranged in the entire frame area, itis possible to readily obtain the locations of each ALF block from theALF block size. Therefore, when the filter block flag is created for allof the ALF blocks within the frame like the related art, it is possibleto readily specify the location of the area corresponding to each filterblock flag.

However, for example, in the case of the multi-slice having a pluralityof slices in the frame, it is conceived that the filtering is performedfor the processing-target slice using the filter block flags of a partof the ALF blocks within the frame. In this case, depending on thelocation of the processing-target slice area (i.e., theprocessing-target slice is selected from any one of a plurality ofslices within the frame), the location of the area corresponding to theused filter block flag is different.

However, according to the method in the related art, the location of thefilter block flag or the processing-target slice is not specified.Therefore, control by the filter block flag created in the controlinformation creation unit 112 may not be appropriately operated, and theadaptive filtering may not be appropriately performed.

As described above, since the adaptive filtering unit 113 specifies thelocation of the processing-target slice area which is a part of the areawithin the frame and the location of the area corresponding to thefilter block flag of a part of the ALF block within the frame, it ispossible to appropriately perform the adaptive filtering. That is, sincethe filter block flag that does not affect the processing target sliceof the ALF block which does not include the processing-target slice areais not necessary, the adaptive filtering unit 113 can suppressdegradation of the encoding efficiency of the image compressioninformation.

In addition, since the control information creation unit 112 creates thefilter block flag only for the ALF block which includes theprocessing-target slice area as described above, it is possible tosuppress creation of an unnecessary filter block flag and suppressdegradation of the encoding efficiency of the image compressioninformation.

In addition, the adaptive filtering unit 113 can readily specify thelocation of the processing-target slice area and the location of thearea corresponding to the filter block flag by using the same method asthat of the control information creation unit 112 as described above.

In addition, since the reversible encoder 106 adds the block informationincluding the ALF block size and the filter block flag to the encodingdata (e.g., embedding it into the slice header), the image decodingapparatus which decodes those encoding data can also perform thefiltering based on the block information similar to that of the adaptivefiltering unit 113.

Here, “add” means that the block information is correlated with theencoding data in any type. For example, the block information may bedescribed as the syntax of the encoding data or may be described as userdata. In addition, the block information may be linked with the encodingdata as metadata. That is, “add” includes “embed”, “describe”,“multiplex”, “link”, or the like.

By performing the encoding accompanying with the block informationcreation process or the adaptive filtering as described above, onlythose in which the filter block flag is necessary can be included in theimage compression information. Therefore, the image encoding apparatus100 can suppress degradation of the encoding efficiency by locallycontrolling the filtering during the encoding or the decoding. Forexample, even when each frame of the image is divided into a pluralityof slices, and the encoding using the adaptive filter is performed foreach slice and output, the image encoding apparatus 100 can suppressdegradation of the encoding efficiency.

2. Second Embodiment

Device Configuration

Next, an image decoding apparatus corresponding to the image encodingapparatus 100 of the first embodiment of the invention will bedescribed. FIG. 14 is a block diagram illustrating a configurationexample of the image decoding apparatus as an image processing apparatusaccording to an embodiment of the invention.

The image decoding apparatus 200 decodes the image compressioninformation output from the image encoding apparatus 100 to create adecoding image.

The image decoding apparatus 200 includes an accumulation buffer 201, areversible decoding unit 202, an inverse quantization unit 203, aninverse orthogonal conversion unit 204, a computation unit 205, and adeblock filter 206. In addition, the image decoding apparatus 200 has anadaptive filtering unit 207. Furthermore, the decoding apparatus 200 hasa picture sorting buffer 208 and a D/A (Digital-to-Analog 1) conversionunit 209. Still furthermore, the image decoding apparatus 200 has aframe memory 210, an intra-prediction unit 211, a motion compensationunit 212, and a selector 213.

The accumulation buffer 201 accumulates transmitted image compressioninformation. The reversible decoding unit 202 decodes the informationencoded by the reversible encoder 106 of FIG. 1, supplied from theaccumulation buffer 201, using a decoding scheme corresponding to theencoding scheme of the reversible encoder 106.

When the corresponding macroblock is intra-coded, the reversibledecoding unit 202 decodes the intra-prediction mode information storedin the header of the image compression information and transmits theinformation to the intra-prediction unit 211. In addition, when thecorresponding macroblock is inter-coded, the reversible decoding unit202 decodes the motion vector information stored in the header of theimage compression information and transmits the information to themotion compensation unit 212.

In addition, the reversible decoding unit 202 extracts controlinformation (control information created by the control informationcreation unit 112) for the adaptive filter from the slice header of theimage compression information, decodes it, and supplies that informationto the adaptive filtering unit 207.

The inverse quantization unit 203 performs inverse quantization for theimage decoded by the reversible decoding unit 202 using a schemecorresponding to the quantization scheme of the quantization unit 105 ofFIG. 1. The inverse orthogonal conversion unit 204 performs inverseorthogonal conversion for the output of the inverse quantization unit203 using a scheme corresponding to the orthogonal conversion scheme ofthe orthogonal conversion unit 104 of FIG. 1.

The computation unit 205 adds the prediction image supplied from theselector 213 to the difference information subjected to the inverseorthogonal conversion to create a decoding image. The deblock filter 206removes block distortion of the decoding image created by such anaddition.

The adaptive filtering unit 207 performs filtering for the imagesupplied from the deblock filter 206 based on the information such asthe ALF block size, the filter block flag, and the filter coefficientincluded in the control information supplied from the reversibledecoding unit 202. The adaptive filtering unit 207 performs the sameadaptive filtering as that of the adaptive filtering unit 113 of FIG. 1.As a result, the adaptive filtering unit 207 can reduce block distortionor quantization distortion that was not removed by the deblock filter206.

The adaptive filtering unit 207 supplies the image after the filteringto the frame memory 210, accumulates the image as the reference imageinformation, and outputs it to the picture sorting buffer 208.

The picture sorting buffer 208 performs sorting for the image. That is,the sequence of the frames sorted in the encoding sequence of thepicture sorting buffer 102 of FIG. 1 is sorted in the original displaysequence. The D/A conversion unit 209 performs D/A conversion for theimage supplied from the picture sorting buffer 208 and outputs it. Forexample, the D/A conversion unit 209 outputs the output signal obtainedby the D/A conversion to a display unit (not shown) to display it.

When the corresponding frame is intra-coded, the intra-prediction unit211 creates a prediction image based on the information supplied fromthe reversible decoding unit 202 and outputs the created predictionimage to the selector 213.

When the corresponding frame is inter-coded, the motion compensationunit 212 performs a motion compensation process for the reference imageinformation stored in the frame memory 210 based on the motion vectorinformation supplied from the reversible decoding unit 202.

When the corresponding macroblock is intra-coded, the selector 213 isconnected to the intra-prediction unit 211 and supplies the imagesupplied from the intra-prediction unit 211 as a prediction image to thecomputation unit 205. In addition, when the corresponding macroblock isinter-coded, the selector 213 is connected to the motion compensationunit 212 and supplies the image supplied from the motion compensationunit 212 as a prediction image to the computation unit 205.

Process Flow

An example of the decoding process flow executed by the image decodingapparatus 200 will be described with reference to the flowchart of FIG.15.

In step S201, the accumulation buffer 210 accumulates the transmittedimage. In step S202, the reversible decoding unit 202 decodes thecompression image supplied from the accumulation buffer 201. That is,the I-picture, the P-picture, and the B-picture are encoded by thereversible encoder 106 of FIG. 1.

In this case, the motion vector information, the reference frameinformation, the prediction mode information (information representingthe intra-prediction mode or the inter-prediction mode), or the like arealso decoded.

That is, when the prediction mode information is intra-prediction modeinformation, the prediction mode information is supplied to theintra-prediction unit 211. When the prediction mode information is theinter-prediction mode information, the reference frame information andthe motion vector information corresponding to the prediction modeinformation are supplied to the motion compensation unit 212.

Furthermore, the reversible decoding unit 202 extracts the controlinformation for the adaptive filtering from the slice header of theimage compression information in step S202 and decode it. The decodedcontrol information is supplied to the adaptive filtering unit 207.

In step S204, the inverse quantization unit 203 performs inversequantization for the conversion coefficient decoded in step S202 usingthe characteristic corresponding to the characteristic of thequantization unit 105 of FIG. 1. In step S205, the inverse orthogonalconversion unit 204 performs inverse orthogonal conversion for theconversion coefficient subjected to the inverse quantization in stepS204 using the characteristic corresponding to the characteristic of theorthogonal conversion unit 104 of FIG. 1. As a result, differenceinformation corresponding to the input of the orthogonal conversion unit104 (the output of the computation unit 103) of FIG. 1 is decoded.

In step S206, the computation unit 205 adds the prediction imageselected in step S212 described below to the difference information. Asa result, the original image is decoded. In step S207, the deblockfilter 206 performs filtering for the image output from the computationunit 205. As a result, block distortion is removed.

In step S208, the adaptive filtering unit 207 further performs adaptivefiltering for the image subjected to the deblock filtering. Thisadaptive filtering is similar to the process performed by the adaptivefiltering unit 113 of FIG. 1. That is, this adaptive filtering isperformed in a similar way to that described with reference to theflowchart of FIG. 13 using the control information supplied from thereversible decoding unit 202. However, the control information suppliedfrom the reversible decoding unit 202 is also created by the controlinformation creation unit 112 of FIG. 1 and is substantially the same asthe control information supplied from the control information creationunit 112 used by the adaptive filtering unit 113 of FIG. 1.

Through this adaptive filtering, it is possible to reduce blockdistortion or quantization distortion that was not removed by thedeblock filtering.

In step S209, the frame memory 210 stores the image subjected to thefiltering.

When the intra-prediction mode information is supplied, theintra-prediction unit 211 performs intra-prediction for theintra-prediction mode in step S210. In addition, when theinter-prediction mode information is supplied, the motion compensationunit 212 performs motion compensation for the inter-prediction mode instep S211.

In step S212, the selector 213 selects a prediction image. That is, theselector 213 selects any one of the prediction image created by theintra-prediction unit 211 or the prediction image created by the motioncompensation unit 212 and supplies the selected prediction image to thecomputation unit 205.

For example, in the case of the intra-coded image, the selector 213selects the prediction image created by the intra-prediction unit 211and supplies it to the computation unit 205. In addition, in the case ofthe inter-coded image, the selector 213 selects the prediction imagecreated by the motion compensation unit 212 and supplies it to thecomputation unit 205.

In step S213, the picture sorting buffer 208 performs sorting. That is,the frame sequence sorted for the encoding by the picture sorting buffer102 of the image encoding apparatus 100 of FIG. 1 is sorted in theoriginal display sequence.

In step S214, the D/A conversion unit 209 performs D/A conversion forthe image from the picture sorting buffer 208. This image is output to adisplay unit (not shown) and displayed thereon.

As such, in the image decoding apparatus 200, the reversible decodingunit 202 extracts the control information supplied from the imageencoding apparatus 100 and decodes it, and the adaptive filtering unit207 performs adaptive filtering in a similar way to that of the adaptivefiltering unit 113 of the image encoding apparatus 100 using suchcontrol information.

As described above, by performing the adaptive filtering, it is possibleto allow the adaptive filtering unit 207 to appropriately performfiltering demanded by the processing-target slice of a plurality ofslices formed in the frame based on the filter block flag of a part ofthe ALF block within the frame. As a result, the adaptive filtering unit207 can remove block distortion or quantization distortion, that was notremoved by the deblock filter, in the processing-target slice.

That is, similar to the case of the adaptive filtering unit 113, theadaptive filtering unit 207 can appropriately perform filtering for theprocessing-target slice based on the filter block flag supplied only tothe ALF block demanded by the processing-target slice.

Therefore, the image decoding apparatus 200 can appropriately decode theimage compression information including only the filter block flagdemanded by the image encoding apparatus 100. That is, the imagedecoding apparatus 200 can suppress degradation of the encodingefficiency by locally controlling the filtering during the encoding orthe decoding. For example, even when each frame of the image is dividedinto a plurality of slices, and the encoded image compressioninformation is decoded for each slice using the adaptive filter andoutputted, it is possible to suppress degradation of the encodingefficiency.

3. Third Embodiment

Another Example of Processing-target ALF Block

In the aforementioned descriptions, the control information creationunit 112 creates the filter block flag for all of the ALF blocks atleast including the processing-target slice area and performs filteringfor all of the ALF blocks at least including the processing-target slicearea adaptive filtering unit 113.

However, for example, when only a single pixel of the processing-targetslice area is included in the ALF block, the filtering hardly affectsthe image quality of the processing-target slice. As such, even when thefiltering is performed for the ALF blocks having a small ratio of theprocessing-target slice area, a sufficient effect may not be obtained,and the processing (load) may become useless.

In this regard, only the ALF blocks including a predetermined ratio ormore of the processing-target slice areas may be set to the processingtarget. This predetermined ratio functioning as a threshold value may bearbitrarily set. In addition, this value may be set previously or may beset to change depending on the contents of the image or the like.

Process Flow

An example of a block information creating process flow in this casewill be described with reference to the flowchart of FIG. 16. Theflowchart of FIG. 16 corresponds to the flowchart of FIG. 12.

That is, as shown in FIG. 16, even in this case, the block informationcreation process is performed basically in a similar way to the casedescribed with reference to FIG. 12.

Therefore, the configuration of the control information creation unit112 in this case is similar to the case of FIG. 3.

Each process of steps S351 to S355 of FIG. 16 is performed in a similarway to each process of steps S151 to S155 of FIG. 12.

However, in FIG. 16, as the processing-target ALF block area isspecified, the determination unit 144 determines whether or not apredetermined ratio or more of the processing-target ALF block areas areincluded in the processing-target slice area in step S356.

If it is determined that a predetermined ratio or more of theprocessing-target ALF block areas are included in the processing-targetslice area, the process advances to step S357. In addition, in stepS356, if it is determined that the processing-target ALF blocks includedin the processing-target slice area are smaller than a predeterminedratio, the process advances to step S359.

Each process of steps S357 to S359 is performed in a similar way to eachprocess of steps S157 to S159 in FIG. 12.

As such, the condition for creating the filter block flag may be set toa substantially more useful range in comparison with the firstembodiment. As a result, the image encoding apparatus 100 and the imagedecoding apparatus 200 can further suppress degradation of the encodingefficiency.

In addition, while in the aforementioned descriptions, the encoding orthe decoding using the adaptive filter is performed on a slice-by-slicebasis, the present invention is not limited thereto. The encoding or thedecoding may be performed in more accurate data unit if it is smallerthan frame unit.

4. Fourth Embodiment

Description of QALF

As disclosed in T. Chujoh, N. Wada and G. Yasuda, “Quadtree-basedAdaptive Loop Filter,” ITU-T SG16 Q6 VCEG Contribution, VCEG-AK22(r1),Japan, April, 2009, the ALF block may have a quad-tree structure. Thistechnique is called a quadtree-based adaptive loop filter (QALF). Inthis quadtree structure, a single ALF block area in the upper layer isdivided into four layers in the lower layer.

FIGS. 17A to 17D illustrate an example of designating the filter blockflag in each ALF block by representing the ALF block division using aquadtree structure having a maximum layer number of 3.

FIG. 17A illustrates a layer 0 which is an ALF block functioning as abasis of the quadtree structure. In the quadtree structure, each ALFblock has a block partitioning flag which represents whether or not eachALF block is quad-divided in the lower layer. A value of the blockpartitioning flag of the ALF block shown in FIG. 17A is set to “1”. Thatis, this ALF block is quad-divided in the lower layer (the layer 1).FIG. 17B shows the layer 1. That is, four ALF blocks are provided in thelayer 1.

When the block partitioning flag is set to “0”, the ALF block is notquad-divided in the lower layer. That is, there is no more division, andthe filter block flag is created for that ALF block. That is, the ALFblock having the block partitioning flag having a value of “0” also hasthe filter block flag. Out of “0-1” shown in FIG. 17B, the left side “0”represents the block partitioning flag of the corresponding ALF block,and the right side “1” represents the filter block flag of thecorresponding ALF block.

Two ALF blocks having the block partitioning flag in the layer 1 set to“1” are quad-divided in the further lower layer (the layer 2). FIG. 17Cshows the layer 2. That is, 10 ALF blocks are created in the layer 2.

Similarly, the filter block flag may also be assigned to the ALF blockhaving a block partitioning flag set to “0” in the layer 2. In FIG. 17C,the block partitioning flag of a single ALF block is set to “1”. Thatis, that ALF block is quad-divided in the still further lower layer (thelayer 3). FIG. 17D shows the layer 3. That is, 13 ALF blocks are createdin the layer 3.

The final configuration of the ALF block obtained by thequadtree-dividing as shown in FIGS. 17A to 17D is illustrated in FIG.18. As such, in the quadtree structure, the ALF block size is differentin each layer. That is, the ALF block may have a different size withinthe frame by using the quadtree structure.

The control of the filter block flag in each ALF block is similar tothat of the first embodiment. That is, when a value of the filter blockflag is “0”, the filtering is not performed for the ALF block area(hatched in FIG. 18).

The problem that the encoding efficiency may be degraded in amulti-slice case is also generated when the ALF block is established inthe slice area smaller than the frame. Therefore, the same problem isgenerated in the QALF structure obtained by improving the representationof the ALF block.

FIG. 19 illustrates an example of encoding the area of the slice 1 ofFIG. 5 using the QALF technique. Here, the area encoded by the bold line421 represents the area of the slice 1. Similar to the BALF describedaccording to the first embodiment of the invention, the ALF blockhatched in FIG. 19 does not include the area of the slice 1 and thusbecomes an unnecessary ALF block.

As in the related art, if the filter block flag is provided even in theunnecessary ALF block, the image compression information unnecessarilyincreases, and the encoding efficiency may be degraded.

Even in the case of the QALF, the image encoding apparatus 100 cansuppress the unnecessary filter block flag from being created andincluded in the slice header using the same way as that of the BALFdescribed according to the first embodiment.

That is, in the image encoding apparatus 100 and the image decodingapparatus 200, it is recognized that the block hatched in FIG. 19 doesnot include the corresponding slice (the slice 1 in FIG. 19). In thisregard, the image encoding apparatus 100 creates the filter block flagfor the ALF block having a quadtree structure including thecorresponding slice and supplies it to the image decoding apparatus 200together with the image compression information. Therefore, it ispossible to suppress creation of an unnecessary filter block flag anddegradation of the encoding efficiency.

5. Fifth Embodiment

Personal Computer

A series of the aforementioned processes may be executed in software aswell as hardware. In this case, for example, the present invention maybe embodied in a personal computer as shown in FIG. 20.

In FIG. 20, the CPU 501 of the personal computer 500 executes variousprocesses according to a program stored in a ROM (Read Only Memory) 502or a program loaded on a RAM (Random Access Memory) 503 from a storageunit 513. The RAM 503 appropriately stores data demanded to executevarious processes in the CPU 501.

The CPU 501, the ROM 502, and the RAM 503 are connected to one anothervia a bus 504. The bus 504 is also connected to the input/outputinterface 510.

The input/output interface 510 is connected to an input unit 511 such asa keyboard and a mouse, a display unit such as a CRT (Cathode Ray Tube)and an LCD (Liquid Crystal Display), an output unit 512 such as aloudspeaker, a storage unit 513 such as a hard disc, and a communicationunit 514 such as a modem. The communication unit 514 performs acommunication process via networks including the Internet.

The input/output interface 510 is connected to a drive 515 if necessary,and a removable media 521 such as a magnetic disc, an optical disc, anoptical-magnetic disc, or a semiconductor memory is appropriatelyinstalled, so that a computer program read therefrom is installed in thestorage unit 513 if necessary.

When a series of the aforementioned processes are executed in software,the program constituting the software is installed from a network or arecording medium.

For example, the recording medium may include a removable medium 521such as a magnetic disc (including a flexible disc), an optical disc(including a CD-ROM (Compact Disc-Read Only Memory), a DVD (DigitalVersatile Disc), etc), an optical magnetic disc (MD(Mini Disc)), or asemiconductor memory that records a program distributed to deliver aprogram to users, separate from a device mainframe as shown in FIG. 20,a ROM 502 delivered to a user while in advance assembled in a devicemainframe for storing a program, a hard disc included in the storageunit 513, or the like.

The program executed by the computer may be processed in a time seriesaccording to the sequence described herein or processed in parallel orin a necessary timing, for example, when it is called.

Herein, the steps for describing the program stored in the recordingmedium may include processes executed in a time series according to thedescribed sequence or executed in parallel or individually without beingprocessed in a time series.

In addition, herein, a system refers to the entire apparatus including aplurality of devices (apparatuses).

In the aforementioned descriptions, a configuration described as asingle apparatus (or a processing unit) may be made of a plurality ofapparatuses (or processing units). On the contrary, the configurationdescribed as a plurality of apparatuses (or processing units) may bemade of a single apparatus (or a processing unit). In addition, otherconfigurations than those described above may be added to theconfiguration of each apparatus (or each processing unit). Furthermore,if the configurations or the operations are substantially the same froma systematic viewpoint, a part of the configuration of any apparatus (orprocessing unit) may be included in the configurations of otherapparatuses (or other processing units). That is, embodiments of theinvention are not limited to those described above, but may be variouslychanged without departing from the spirit and the scope of theinvention.

For example, the image encoding apparatus 100 or the image decodingapparatus 200 described above may be applied to any electronic device.Hereinafter, an example thereof will be described.

6. Sixth Embodiment

Television Set

FIG. 21 is a block diagram illustrating an example of a mainconfiguration of a television set which uses the image decodingapparatus 200 according to an embodiment of the invention.

The television set 1000 shown in FIG. 21 includes a terrestrial tuner1013, a video decoder 1015, a video signal processing circuit 1018, agraphics generation circuit 1019, a panel driving circuit 1020, and adisplay panel 1021.

The terrestrial tuner 1013 receives a broadcasting wave signal of aterrestrial analog broadcasting via an antenna, demodulates it to obtaina video signal, and supplies it to the video decoder 1015. The videodecoder 1015 performs a decoding process for the video signal suppliedfrom the terrestrial tuner 1013 and supplies the resultant digitalcomponent signal to the video signal processing circuit 1018.

The video signal processing circuit 1018 performs a predeterminedprocess such as noise removal for the video data supplied from the videodecoder 1015 and supplies the resultant video data to the graphicsgeneration circuit 1019.

The graphics generation circuit 1019 generates video data of a programdisplayed on the display panel 1021 or image data obtained through aprocess based on an application supplied via a network and supplies thegenerated video or image data to the panel driving circuit 1020. Inaddition, the graphics generation circuit 1019 appropriately performs aprocess for generating video data (graphics) for displaying a windowused by a user to select an item and supplying video data obtained byoverlapping the graphics with video data of a program to the paneldriving circuit 1020.

The panel driving circuit 1020 drives the display panel 1021 based onthe data supplied from the graphics generation circuit 1019 and displaysa program video or various windows described above on the display panel1021.

The display panel 1021 is made of an LCD (Liquid Crystal Display) todisplay a program video under control of the panel driving circuit 1020.

In addition, the television set 1000 also includes an audio A/D(Analog/Digital) conversion circuit 1014, an audio signal processingcircuit 1022, an echo cancellation/audio synthesis circuit 1023, anaudio amplification circuit 1024, and a loudspeaker 1025.

The terrestrial tuner 1013 obtains an audio signal as well as a videosignal by demodulating the received broadcasting wave signal. Theterrestrial tuner 1013 supplies the obtained audio signal to the audioA/D conversion circuit 1014.

The audio A/D conversion circuit 1014 performs an A/D conversion processfor the audio signal supplied from the terrestrial tuner 1013 andsupplies the resultant digital audio signal to the audio signalprocessing circuit 1022.

The audio signal processing circuit 1022 performs a predeterminedprocess such as noise removal for the audio data supplied from the audioA/D conversion circuit 1014 and supplies the resultant audio data to theecho cancellation/audio synthesis circuit 1023.

The echo cancellation/audio synthesis circuit 1023 supplies the audiodata supplied from the audio signal processing circuit 1022 to the audioamplification circuit 1024.

The audio amplification circuit 1024 performs a D/A conversion processand an amplification process for the audio data supplied from the echocancellation/audio synthesis circuit 1023 to adjust the audio to apredetermined volume and outputs the audio to the loudspeaker 1025.

Furthermore, the television set 1000 also has a digital tuner 1016 andan MPEG decoder 1017.

The digital tuner 1016 receives a digital broadcasting wave signal (suchas a terrestrial digital broadcasting or a BS (BroadcastingSatellite)/CS(Communications Satellite) digital broadcasting) via anantenna, demodulates it to obtain an MPEG-TS (Moving Picture ExpertsGroup-Transport Stream), and supplies it to the MPEG decoder 1017.

The MPEG decoder 1017 releases a scramble applied to the MPEG-TSsupplied from the digital tuner 1016 to extract program data streams ofa reproduction target (a watching target). The MPEG decoder 1017 decodesaudio packets included in the extracted streams and supplies theresultant audio data to the audio signal processing circuit 1022. At thesame time, the MPEG decoder 1017 decodes the video packets included inthe streams and supplies the resultant video data to the video signalprocessing circuit 1018. The MPEG decoder 1017 supplies the EPG(Electronic Program Guide) data extracted from the MPEG-TS to the CPU1032 via a path (not shown).

The television set 1000 uses the aforementioned image decoding apparatus200 as the MPEG decoder 1017 for decoding video packets as describedabove. In addition, the MPEG-TS transmitted from a broadcasting centeror the like is encoded by the image encoding apparatus 100.

Similar to the image decoding apparatus 200, the MPEG decoder 1017appropriately decodes the image compression information including onlythe filter block flag necessary for the image encoding apparatus 100.Therefore, it is possible to suppress degradation of the encodingefficiency by locally controlling the filtering. For example, even wheneach frame of the image is divided into a plurality of slices, and theimage compression information encoded for each slice is decoded usingthe adaptive filter to output it, it is possible to suppress degradationof the encoding efficiency.

Similar to the video data supplied from the video decoder 1015, thevideo data supplied from the MPEG decoder 1017 is subjected to apredetermined process in the video signal processing circuit 1018,appropriately overlapped with the video data generated in the graphicsgeneration circuit 1019, and supplied to the display panel 1021 via thepanel driving circuit 1020 so that the image is displayed.

Similar to the audio signal supplied from the audio A/D conversioncircuit 1014, the audio data supplied from the MPEG decoder 1017 issubjected to a predetermined process in the audio signal processingcircuit 1022, supplied to a the audio amplification circuit 1024 via theecho cancellation/audio synthesis circuit 1023, and subjected to a D/Aconversion process or an amplification process. As a result, audioadjusted to a predetermined volume is output from the loudspeaker 1025.

The television set 1000 also has a microphone 1026 and an A/D conversioncircuit 1027.

The A/D conversion circuit 1027 receives a user's voice through amicrophone 1026 provided in the television set 1000 for voicecommunication, performs an A/D conversion process for the received voicesignal, and supplies the resultant digital audio data to the echocancellation/audio synthesis circuit 1023.

When the voice data of a user (user A) of the television set 1000 aresupplied from the A/D conversion circuit 1027, the echocancellation/audio synthesis circuit 1023 performs echo cancellation forthe voice data of a user A, synthesizes them with another audio data,and supplies the resultant audio data to the loudspeaker 1025 via theaudio amplification circuit 1024.

Furthermore, the television set 1000 also includes an audio codec 1028,an internal bus 1029, an SDRAM (Synchronous Dynamic Random AccessMemory) 1030, a flash memory 1031, a CPU 1032, an USB (Universal SerialBus) I/F 1033, and a network I/F 1034.

The A/D conversion circuit 1027 receives a user's voice signal receivedby the microphone 1026 provided in the television set 1000 for voicecommunication, performs an A/D conversion process for the received voicesignal, and supplies the resultant digital audio data to the audio codec1028.

The audio codec 1028 converts the audio data supplied from the A/Dconversion circuit 1027 into data having a predetermined format fortransmitting via a network and supplies the data to the network I/F 1034via an internal bus 1029.

The network I/F 1034 is connected to a network via a cable installed inthe network terminal 1035. The network I/F 1034 transmits the audio datasupplied from the audio codec 1028, for example, to another deviceconnected to that network. In addition, the network I/F 1034 receivesthe audio data, for example, transmitted from another device connectedvia a network, through a network terminal 1035 and supplies the audiodata to the audio codec 1028 through the internal bus 1029.

The audio codec 1028 converts the audio data supplied from the networkI/F 1034 into the data having a predetermined format and supplies thedata to the echo cancellation/audio synthesis circuit 1023.

The echo cancellation/audio synthesis circuit 1023 performs echocancellation for the audio data supplied from the audio codec 1028 andoutputs the audio data obtained by synthesizing them with another audiodata to the loudspeaker 1025 via the audio amplification circuit 1024.

The SDRAM 1030 stores various data necessary for the CPU 1032 to performprocesses.

The flash memory 1031 stores a program executed by the CPU 1032. Aprogram stored in the flash memory 1031 is read by the CPU 1032 at apredetermined timing such as when the television set 1000 is operated.The flash memory 1031 also stores the EPG data obtained through adigital broadcasting or the data obtained from a predetermined servervia a network.

For example, the flash memory 1031 stores the MPEG-TS including contentsdata obtained from a predetermined server via a network under control ofthe CPU 1032. The flash memory 1031 supplies the MPEG-TS to the MPEGdecoder 1017 via an internal bus 1029, for example, under control of theCPU 1032.

Similar to the MPEG-TS supplied from the digital tuner 1016, the MPEGdecoder 1017 processes the MPEG-TS. As such, the television set 1000 mayreceive contents data including the video and the audio via a network.The contents data may be decoded using the MPEG decoder 1017, and thevideo or the audio may be displayed or output.

In addition, the television set 1000 also includes an optical receiver1037 for receiving an infrared signal transmitted from the remotecontroller 1051.

The optical receiver 1037 receives the infrared light from the remotecontroller 1051 and outputs the control code representing a user'smanipulation obtained by decoding the light to the CPU 1032.

The CPU 1032 executes the program stored in the flash memory 1031 andcontrols the entire operation of the television set 1000 according tothe control code supplied from the optical receiver 1037. Each componentof the television set 1000 and the CPU 1032 are connected to one anothervia a path (not shown).

The USB I/F 1033 transmits/receives data to/from a device outside thetelevision set 1000 connected via a USB cable installed in the USBterminal 1036. The network I/F 1034 is connected to a network via acable installed in the network terminal 1035 and also transmits/receivesdata other than the audio data to/from various devices connected to anetwork.

The television set 1000 can appropriately decode the image compressioninformation including only the filter block flag demanded by the imageencoding apparatus 100 by using the image decoding apparatus 200 as theMPEG decoder 1017. As a result, the television set 1000 can suppressdegradation of the encoding efficiency for locally controlling thefiltering for the broadcasting signal received via an antenna or thecontents data obtained via a network.

7. Seventh Embodiment

Mobile Phone

FIG. 22 is a block diagram illustrating an example of a mainconfiguration of the mobile phone used in the image encoding apparatusand the image decoding apparatus according to an embodiment of theinvention.

The mobile phone 1100 shown in FIG. 22 includes a main control unit 1150configured to collectively control each component, a power circuit unit1151, a manipulation input control unit 1152, an image encoder 1153, acamera I/F unit 1154, an LCD control unit 1155, an image decoder 1156, amultiplexer/demultiplexer unit 1157, a recording reproduction unit 1162,a modulation/demodulation circuit unit 1158, and an audio codec 1159.These units are connected to one another via a bus 1160.

In addition, the mobile phone 1100 includes a manipulation key 1119, aCCD (Charge Coupled Devices) camera 1116, a liquid crystal display 1118,a storage unit 1123, a transceiver circuit 1163, an antenna 1114, amicrophone 1121, and a loudspeaker 1117.

When a call-clearing and power key is turned on by a user'smanipulation, the power circuit unit 1151 drives the mobile phone 1100to an operable state by supplying power to each component from a batterypack.

The mobile phone 1100 performs various operations such astransmitting/receiving audio signals, electronic mails, or image data,taking an image, and recording data using various modes such as a voicecalling mode or a data communication mode based on the control of themain control unit 1150 including a CPU, a ROM, a RAM, or the like.

For example, in the voice calling mode, the mobile phone 1100 converts avoice signal collected from the microphone 1121 into digital audio datausing the audio codec 1159, performs a spectrum diffusion process usingthe modulation/demodulation circuit unit 1158, and performs adigital-analog conversion process and a frequency conversion processusing the transceiver circuit 1163. The mobile phone 1100 transmits thetransmission signal obtained by those conversion processes to a basestation (not shown) via an antenna 1114. The transmission signal (audiosignal) transmitted to the base station is supplied to the mobile phoneof a called party via a public switched telephone network.

For example, in the voice calling mode, the mobile phone 1100 amplifiesthe receive signal received by the antennal 1114 using the transceivercircuit 1163, performs a frequency conversion process and ananalog-digital conversion process, performs a spectrum inverse diffusionprocess using the modulation/demodulation circuit unit 1158, andconverts it into the analog audio signal using the audio codec 1159. Themobile phone 1100 outputs the analog audio signal resulting from theconversion to the loudspeaker 1117.

For example, when an electronic mail is transmitted in a datacommunication mode, the mobile phone 1100 receives electronic mail textdata input by the manipulation of the manipulation key 1119 using themanipulation input control unit 1152. The mobile phone 1100 processesthe text data using the main control unit 1150 and displays the data asan image on the liquid crystal display 1118 using the LCD control unit1155.

The mobile phone 1100 generates the electronic mail data based on auser's instruction or the text data received by the manipulation inputcontrol unit 1152 using the main control unit 1150. The mobile phone1100 performs a spectrum diffusion process for the electronic mail datausing the modulation/demodulation circuit unit 1158 and performs adigital-analog conversion process and a frequency conversion processusing the transceiver circuit 1163. The mobile phone 1100 transmits thetransmission signal obtained through those conversion processes to abase station (not shown) via an antenna 1114. The transmission signal(electronic mail) transmitted to the base station is supplied to apredetermined destination via a network, a mail server, or the like.

For example, when the electronic mail is received in the datacommunication mode, the mobile phone 1100 receives the signaltransmitted from the base station using the transceiver circuit 1163 viathe antenna 1114, amplifies the signal, and further performs a frequencyconversion process and an analog-digital conversion process. The mobilephone 1100 performs a spectrum inverse diffusion process for thatreceive signal using the modulation/demodulation circuit unit 1158 torestore the original electronic mail data. The mobile phone 1100displays the restored electronic mail data on the liquid crystal display1118 using the LCD control unit 1155.

In addition, the mobile phone 1100 may record the received electronicmail data in the storage unit 1123 using the recording reproduction unit1162.

This storage unit 1123 is an arbitrary rewritable storage medium. Thestorage unit 1123 may include, for example, a semiconductor memory suchas a RAM or an internal flash memory, a hard disc, or a removable mediumsuch as a magnetic disc, an optical magnetic disc, an optical disc, aUSB memory, or a memory card. Needless to say, other storage mediums maybe used.

For example, when the image data is transmitted in the datacommunication mode, the mobile phone 1100 creates image data bycapturing an image using a CCD camera 1116. The CCD camera 1116 has anoptical device such as a lens or an aperture and a CCD as anphotoelectric conversion element. The CCD camera 1116 captures an imageof an object, converts an intensity of the received light into anelectric signal, and creates image data of an image of the object. TheCCD camera 1116 encodes the image data using the image encoder 1153through the camera I/F unit 1154 to convert the image data into theencoded image data.

The mobile phone 1100 uses the image encoding apparatus 100 as an imageencoder 1153 for executing these processes. Therefore, similar to theimage encoding apparatus 100, the image encoder 1053 can suppressdegradation of the encoding efficiency by locally controlling thefiltering. For example, even when each frame of the image is dividedinto a plurality of slices, and each slice is encoded using the adaptivefilter and output, the image encoder 1053 can suppress degradation ofthe encoding efficiency.

In addition, at the same time, the mobile phone 1100 analog-to-digitalconverts the audio signal collected by the microphone 1121 during theimage capturing of the CCD camera 1116 using the audio codec 1159 andfurther encodes the collected audio signal.

The mobile phone 1100 multiplexes the encoded image data supplied fromthe image encoder 1153 and the digital audio data supplied from theaudio codec 1159 using the multiplexer/demultiplexer unit 1157 accordingto a predetermined scheme. The mobile phone 1100 performs a spectrumdiffusion process for the resultant multiplexed data using themodulation/demodulation circuit unit 1158 and performs a digital-analogconversion process and a frequency conversion process using thetransceiver circuit 1163. The mobile phone 1100 transmits thetransmission signal obtained by the conversion processes to a basestation (not shown) via an antenna 1114. The transmission signal (imagedata) transmitted to the base station is supplied to the called partyvia a network or the like.

In addition, when the image data is not transmitted, the mobile phone1100 may display the image data generated by the CCD camera 1116 on theliquid crystal display 1118 using the LCD control unit 1155 withoutusing the image encoder 1153.

For example, in the data communication mode, when the data of a movingpicture file is linked to a simplified homepage, the mobile phone 1100receives the signal transmitted from the base station via the antenna1114 using the transceiver circuit 1163, amplifies it, and performs afrequency conversion process and an analog-digital conversion process.The mobile phone 1100 performs a spectrum inverse diffusion process forthe receive signal using the modulation/demodulation circuit unit 1158to restore the original multiplexed data. The mobile phone 1100separates the multiplexed data using the multiplexer/demultiplexer unit1157 and divides them into encoded image data and audio data.

The mobile phone 1100 creates reproduction moving picture data bydecoding the encoded image data using the image decoder 1156 anddisplays them on the liquid crystal display 1118 using the LCD controlunit 1155. As a result, for example, the moving picture data included inthe moving picture file linked to the simplified homepage are displayedon the liquid crystal display 1118.

The mobile phone 1100 uses the aforementioned image decoding apparatus200 as the image decoder 1156 for executing such a process. Therefore,similar to the image decoding apparatus 200, the image decoder 1156appropriately decodes the image compression information including onlythe filter block flag demanded by the image encoding apparatus 100.Therefore, as a result, it is possible to suppress degradation of theencoding efficiency by locally controlling the filtering. For example,even when each frame of the image is divided into a plurality of slices,and the image compression information encoded for each slice is decodedusing the adaptive filter and output, it is possible to suppressdegradation of the encoding efficiency.

In this case, the mobile phone 1100 converts the digital audio data intothe analog audio signal using the audio codec 1159 and outputs theanalog audio signal to the loudspeaker 1117. As a result, for example,the audio data included in the moving picture file linked to asimplified homepage is reproduced.

In addition, similar to the electronic mail, the mobile phone 1100 mayrecord (store) the received data linked to a simplified homepage in thestorage unit 1123 using the recording reproduction unit 1162.

In the mobile phone 1100, the main control unit 1150 analyzes the2-dimensional code obtained by the CCD camera 1116 through the imagecapturing so as to obtain the information recorded on the 2-dimensionalcode.

Furthermore, the mobile phone 1100 can communicate with an externaldevice using the infrared communication unit 1181 through an infraredray.

In the mobile phone 1100, by using the image encoding apparatus 100 asthe image encoder 1153, it is possible to suppress degradation of theencoding efficiency caused by locally controlling the filtering, forexample, for the encoded data created by encoding the image datagenerated in the CCD camera 1116. As a result, the mobile phone 1100 canprovide the encoded data (image data) having an excellent encodingefficiency to other devices.

In addition, in the mobile phone 1100, by using the image decodingapparatus 200 as the image decoder 1156, it is possible to appropriatelydecode the image compression information including only the filter blockflag demanded by the image encoding apparatus 100. As a result, in themobile phone 1100, it is possible to suppress degradation of theencoding efficiency for locally controlling the filtering, for example,for the data on a moving picture file linked to a simplified homepage.

While, in the aforementioned descriptions, the CCD camera 1116 is usedin the mobile phone 1100, an image sensor using a CMOS (ComplementaryMetal Oxide Semiconductor) (CMOS image sensor) may be used instead ofthe CCD camera 1116. Even in this case, similar to the CCD camera 1116,the mobile phone 1100 can create image data on the image of an object bycapturing the image of the object.

While, in the aforementioned descriptions, the mobile phone 1100 hasbeen exemplified, the image encoding apparatus 100 and the imagedecoding apparatus 200 may be applied to any other apparatus having animage capturing function or a communication function similar to themobile phone 1100, such as a PDA (Personal Digital Assistants), a smartphone, a UMPC (Ultra Mobile Personal Computer), a net-book, or anotebook type personal computer.

8. Eighth Embodiment

Hard Disc Recorder

FIG. 23 is a block diagram illustrating an example of a mainconfiguration of the hard disc recorder using the image encodingapparatus and the image decoding apparatus according to an embodiment ofthe invention.

The hard disc recorder (HDD recorder) 1200 shown in FIG. 23 is anapparatus for storing, in an internal hard disc, audio data and videodata of a broadcasting program included in the broadcasting wave signal(television signal) transmitted from a satellite or a terrestrialantenna or the like and received by the tuner and providing the storeddata to a user at the timing responding to a user's instruction.

The hard disc recorder 1200 may extract the audio data and the videodata, for example, from the broadcasting signal, appropriately decodethem, and store them in the hard disc. The hard disc recorder 1200 mayobtain the audio data or the video data from other devices, for example,via a network, decode them, and store them in an internal hard disc.

Furthermore, the hard disc recorder 1200 may decode the audio data orthe video data stored, for example, in an internal hard disc, supplythem to a monitor 1260, display the image on a screen of the monitor1260, and output the audio from the loudspeaker of the monitor 1260. Inaddition, the hard disc recorder 1200 may decode, for example, the audiodata and the video data extracted from the broadcasting signal obtainedusing a tuner or the audio data or the video data obtained from otherdevices via a network, supply them to the monitor 1260, display theimage on a screen of the monitor 1260, and output the audio from theloudspeaker of the monitor 1260.

Of course, other operations may be possible.

As shown in FIG. 23, the hard disc recorder 1200 includes a receiverunit 1221, a demodulation unit 1222, a demultiplexer 1223, an audiodecoder 1224, a video decoder 1225, and a recorder control unit 1226.The hard disc recorder 1200 further includes an EPG data memory 1227, aprogram memory 1228, a work memory 1229, a display converter 1230, anOSD (On Screen Display) control unit 1231, a display control unit 1232,a recording reproduction unit 1233, a D/A converter 1234, and acommunication unit 1235.

In addition, the display converter 1230 includes a video encoder 1241.The recording reproduction unit 1233 includes an encoder 1251 and adecoder 1252.

The receiver unit 1221 receives an infrared signal from a remotecontroller (not shown), converts it into an electric signal, and outputsit to the recorder control unit 1226. The recorder control unit 1226 isconfigured of, for example, a microprocessor or the like and executesvarious processes depending on a program stored in the program memory1228. In this case, the recorder control unit 1226 uses the work memory1229 as necessary.

The communication unit 1235 is connected to a network and communicateswith other apparatuses via a network. For example, the communicationunit 1235 is controlled by the recorder control unit 1226 andcommunicates with a tuner (not shown) to usually output a channelselection control signal to the tuner.

The demodulation unit 1222 demodulates the signal supplied from thetuner and outputs it to the demultiplexer 1223. The demultiplexer 1223separates the data supplied from the demodulation unit 1222 into theaudio data, the video data, and the EPG data and outputs them to theaudio decoder 1224, the video decoder 1225, or the recorder control unit1226, respectively.

The audio decoder 1224 decodes the input audio data and outputs them tothe recording reproduction unit 1233. The video decoder 1225 decodes theinput video data and outputs them to the display converter 1230. Therecorder control unit 1226 supplies the input EPG data to the EPG datamemory 1227 to store them therein.

The display converter 1230 encodes the video data supplied from thevideo decoder 1225 or the recorder control unit 1226 into, for example,NTSC (National Television Standards Committee) type video data using thevideo encoder 1241 and outputs it to the recording reproduction unit1233. In addition, the display converter 1230 converts the screen sizeof the video data supplied from the video decoder 1225 or the recordercontrol unit 1226 into a size corresponding to the size of the monitor1260, converts them into the NTSC type video data using the videoencoder 1241, converts them into an analog signal, and outputs them tothe display control unit 1232.

The display control unit 1232 overlaps the OSD signal output from theOSD control unit 1231 with the video signal input from the displayconverter 1230 under control of the recorder control unit 1226 andoutputs it to the display unit of the monitor 1260 to display it.

In addition, the audio data output from the audio decoder 1224 isconverted into the analog signal by the D/A converter 1234 and suppliedto the monitor 1260. The monitor 1260 outputs this audio signal from theinternal loudspeaker.

The recording reproduction unit 1233 has a hard disc as a storage mediumfor recording the video data, the audio data, or the like.

The recording reproduction unit 1233 encodes, for example, the audiodata supplied from the audio decoder 1224 using the encoder 1251. Inaddition, the recording reproduction unit 1233 encodes the video datasupplied from the video encoder 1241 of the display converter 1230 usingthe encoder 1251. The recording reproduction unit 1233 synthesizes theencoded data of the audio data with the encoded data of the video datausing the multiplexer. The recording reproduction unit 1233 performs achannel encoding for the synthesized data, amplifies the data, andwrites the data on the hard disc using a recording head.

The recording reproduction unit 1233 reproduces the data recorded on thehard disc using a reproduction head, amplifies the data, and separatesit into the audio data and the video data using the demultiplexer. Therecording reproduction unit 1233 decodes the audio data and the videodata using the decoder 1252. The recording reproduction unit 1233digital-to-analog converts the decoded audio data and outputs the resultto the loudspeaker of the monitor 1260. In addition, the recordingreproduction unit 1233 digital-to-analog converts the decoded video dataand outputs the result to the display unit of the monitor 1260.

The recorder control unit 1226 reads the most current EPG data from theEPG data memory 1227 based on a user's instruction represented by theinfrared ray signal received by the receiver unit 1221 from the remotecontroller and supplies it to the OSD control unit 1231. The OSD controlunit 1231 generates image data corresponding to the input EPG data andoutputs it to the display control unit 1232. The display control unit1232 outputs the video data input from the OSD control unit 1231 to thedisplay unit of the monitor 1260 and displays it. As a result, the EPG(electronic program guide) is displayed on the display unit of themonitor 1260.

In addition, the hard disc recorder 1200 can obtain various data such asthe video data, the audio data, or the EPG data supplied from otherdevices via a network such as the Internet.

The communication unit 1235 is controlled by the recorder control unit1226 to obtain encoded data such as video data, audio data, and EPG datatransmitted from other devices via a network and supply them to therecorder control unit 1226. The recorder control unit 1226 supplies, forexample, the obtained encoded data of the video data or the audio datato the recording reproduction unit 1233 and stores them in the harddisc. In this case, the recorder control unit 1226 and the recordingreproduction unit 1233 may perform a re-encoding or the like asnecessary.

In addition, the recorder control unit 1226 decodes the encoded data ofthe obtained video data or the audio data and supplies the resultantvideo data to the display converter 1230. Similar to the video datasupplied from the video decoder 1225, the display converter 1230processes the video data supplied from the recorder control unit 1226and supplies the video data to the monitor 1260 using the displaycontrol unit 1232 to display the image.

In addition, in synchronization with the image display, the recordercontrol unit 1226 may supply the decoded audio data to the monitor 1260via the D/A converter 1234 and output the audio from the loudspeaker.

Furthermore, the recorder control unit 1226 decodes the encoded data ofthe obtained EPG data and supplies the decoded EPG data to the EPG datamemory 1227.

The hard disc recorder 1200 described above uses the image decodingapparatus 200 as a decoder embedded in the video decoder 1225, thedecoder 1252, and the recorder control unit 1226. Therefore, similar tothe image decoding apparatus 200, the decoder embedded in the videodecoder 1225, the decoder 1252, and the recorder control unit 1226appropriately decode the image compression information including onlythe filter block flag demanded by the image encoding apparatus 100.Therefore, as a result, it is possible to suppress degradation of theencoding efficiency by locally controlling the filtering. For example,even when each frame of the image is divided into a plurality of slices,and the image compression information encoded for each slice is decodedusing the adaptive filter and output, it is possible to suppressdegradation of the encoding efficiency.

Therefore, the hard disc recorder 1200 can appropriately decode theimage compression information including only the filter block flagdemanded by the image encoding apparatus 100. As a result, the hard discrecorder 1200 can suppress degradation of the encoding efficiency forlocally controlling the filtering, for example, for the video datareceived through the tuner or the communication unit 1235 or the videodata recorded on the hard disc of the recording reproduction unit 1233.

In addition, the hard disc recorder 1200 uses the image encodingapparatus 100 as the encoder 1251. Therefore, similar to the imageencoding apparatus 100, the encoder 1251 can suppress degradation of theencoding efficiency for locally controlling the filtering. For example,even when each frame of the image is divided into a plurality of slices,and each slice is decoded using the adaptive filter and output, theencoder 1251 can suppress degradation of the encoding efficiency.

Therefore, the hard disc recorder 1200 can suppress degradation of theencoding efficiency by locally controlling the filtering, for example,for the encoded data recorded on the hard disc. As a result, the harddisc recorder 1200 can more efficiently use the storage area of the harddisc.

While, in the aforementioned descriptions, the hard disc recorder 1200which records the video data or the audio data on a hard disc has beenexemplified, any other recording medium may be used. For example, evenwhen the recorder uses a recording media such as a flash memory, anoptical disc, or a video tape other than the hard disc, similar to theaforementioned hard disc recorder 1200, the image encoding apparatus 100and the image decoding apparatus 200 can be used.

9. Ninth Embodiment

Camera

FIG. 24 is a block diagram illustrating an example of a mainconfiguration of the camera using the image encoding apparatus and theimage decoding apparatus according to an embodiment of the invention.

The camera 1300 shown in FIG. 24 captures an image of an object anddisplays the image of the object on the LCD 1316 or records it as imagedata on the recording medium 1333.

The lens block 1311 inputs the light (i.e., the image of the object) tothe CCD/CMOS 1312. The CCD/CMOS 1312 is an image sensor using a CCD or aCMOS which converts the intensity of the received light into an electricsignal and supplies it to the camera signal processing unit 1313.

The camera signal processing unit 1313 converts the electric signalsupplied from the CCD/CMOS 1312 into a color difference signal (Y, Cr,and Cb) and supplies it to the image signal processing unit 1314. Theimage signal processing unit 1314 performs a predetermined imageprocessing for the image signal supplied from the camera signalprocessing unit 1313 under control of the controller 1321 or recodes theimage signal using the encoder 1341. The image signal processing unit1314 supplies the encoded data created by encoding the image signal tothe decoder 1315. Furthermore, the image signal processing unit 1314obtains the display data generated in the on-screen display (OSD) 1320and supplies them to the decoder 1315.

In the aforementioned descriptions, the camera signal processing unit1313 appropriately uses the DRAM (Dynamic Random Access Memory) 1318connected via the bus 1317 and stores the encoded data obtained byencoding the image data or the like in the DRAM 1318.

The decoder 1315 decodes the encoded data supplied from the image signalprocessing unit 1314 and supplies the resultant image data (decodedimage data) to the LCD 1316. In addition, the decoder 1315 supplies thedisplay data supplied from the image signal processing unit 1314 to theLCD 1316. The LCD 1316 appropriately synthesizes the image of thedecoded image data supplied from the decoder 1315 with the image of thedisplay data and displays the synthesized image.

The on-screen display 1320 outputs the display data such as a menuwindow or an icon including a symbol, a character, or a figure to theimage signal processing unit 1314 via the bus 1317 under control of thecontroller 1321.

The controller 1321 performs various processes based on the signalrepresenting the content instructed by a user using the manipulationunit 1322 and also controls the image signal processing unit 1314, theDRAM 1318, the external interface 1319, the on-screen display 1320, themedia drive 1323, or the like via the bus 1317. The programs or data orthe like necessary for the controller 1321 to execute various processesare stored in the flash ROM 1324.

For example, the controller 1321 may encode the image data stored in theDRAM 1318 or decode the encoded data stored in the DRAM 1318 instead ofthe image signal processing unit 1314 or the decoder 1315. In this case,the controller 1321 may perform the encoding or the decoding using thesame scheme as the encoding or decoding scheme of the image signalprocessing unit 1314 or the decoder 1315 or may perform the encoding orthe decoding using a scheme different from that of the image signalprocessing unit 1314 or the decoder 1315.

For example, when the initiation of the image printing is instructedfrom the manipulation unit 1322, the controller 1321 reads the imagedata from the DRAM 1318 and supplies the image data to the printer 1334connected to the external interface 1319 via the bus 1317 to print them.

For example, when it is instructed from the manipulation unit 1322 torecord the image, the controller 1321 reads the encoded data from theDRAM 1318 and supplies the encoded data to the recording medium 1333installed in the media drive 1323 via the bus 1317 to store them.

The recording medium 1333 is any readable/writable removable medium suchas a magnetic disc, an optical magnetic disc, an optical disc, or asemiconductor memory. The type of the recording medium 1333 as well asthe type of the removable medium may be arbitrarily selected from a tapedevice, a disc, a memory card, or the like. Of course, a non-contact ICcard may be used.

In addition, the media drive 1323 and the recording medium 1333 may beintegrated into a single body and configured by a non-portable storagemedium such as an internal hard disc drive or an SSD (Solid StateDrive).

When the external interface 1319 is configured of, for example, a USBinput/output terminal or the like to print out the image, the externalinterface 1319 is connected to the printer 1334. In addition, theexternal interface 1319 is connected to the drive 1331 as necessary, anda removable medium 1332 such as a magnetic disc, an optical disc, or anoptical magnetic disc is appropriately installed therein so that thecomputer program read therefrom is installed in the flash ROM 1324 asnecessary.

Furthermore, the external interface 1319 has a network interfaceconnected to a predetermined network such as a LAN or the Internet. Thecontroller 1321 may read the encoded data from the DRAM 1318, forexample, according to an instruction from the manipulation unit 1322 andsupply the encoded data from the external interface 1319 to otherdevices connected via a network. In addition, the controller 1321 mayobtain the encoded data or the image data supplied from other devicesvia a network using an external interface 1319 and store them in theDRAM 1318 or supply them to the image signal processing unit 1314.

The camera 1300 described above uses the image decoding apparatus 200 asthe decoder 1315. Therefore, similar to the image decoding apparatus200, the decoder 1315 appropriately decodes the image compressioninformation including only the filter block flag demanded by the imageencoding apparatus 100. Therefore, as a result, it is possible tosuppress degradation of the encoding efficiency by locally controllingthe filtering. For example, even when each frame of the image is dividedinto a plurality of slices, and the image compression informationencoded for each slice is decoded using the adaptive filter and output,it is possible to suppress degradation of the encoding efficiency.

Therefore, the camera 1300 can appropriately decode the imagecompression information including only the filter block flag demanded bythe image encoding apparatus 100. As a result, the camera 1300 cansuppress degradation of the encoding efficiency for locally controllingthe filtering, for example, for the image data generated in the CCD/CMOS1312, the encoded data of the video data read from the recording medium1333, or the encoded data of the video data obtained via a network.

In addition, the camera 1300 uses the image encoding apparatus 100 asthe encoder 1341. Therefore, similar to the image encoding apparatus100, the encoder 1341 can suppress degradation of the encodingefficiency by locally controlling the filtering. For example, even wheneach frame of the image is divided into a plurality of slices, and eachslice is encoded using the adaptive filter and output, the encoder 1341can suppress degradation of the encoding efficiency.

Therefore, the camera 1300 can suppress degradation of the encodingefficiency by locally controlling the filtering, for example, for theencoded data recorded in the DRAM 1318 or the recording medium 1333 orthe encoded data provided to other devices. As a result, the camera 1300can more efficiently use the storage area of the DRAM 1318 or therecording medium 1333. In addition, the camera 1300 can provide theencoded data (image data) having an excellent encoding efficiency toother devices.

In addition, the decoding method of the image decoding apparatus 200 maybe applied to the decoding process of the controller 1321. Similarly,the encoding method of the image encoding apparatus 100 may be appliedto the encoding of the controller 1321.

In addition, the image data captured by the camera 1300 may be a movingpicture or a still image.

Of course, the image encoding apparatus 100 and the image decodingapparatus 200 may be applied to an apparatus or a system other than theaforementioned apparatus.

In addition, the size of the macroblock is not limited to the 16×16pixel. For example, all sizes of the macroblocks such as the 32×32pixels shown in FIG. 25 can be used.

While, in the aforementioned descriptions, the flag information or thelike is multiplexed (described) in the bit stream, for example, the flagand the image data (or the bit stream) may be transmitted (recorded) inaddition to the multiplexing. An embodiment in which the flag is linked(added) to the image data (or bit stream) may be possible.

The linking (addition) refers to a state that the image data (or the bitstream) and the flag are linked to each other, and a relationship oftheir physical locations is arbitrarily set. For example, the image data(or the bit stream) and the flag may be transmitted to differenttransmission lines. In addition, the image data (or the bit stream) andthe flag may be recorded in different recording media (or differentrecording area within the same recording medium). In addition, a unitfor linking between the image data (or the bit stream) and the flag maybe arbitrarily set and, for example, may be set to an encoding unit(such as a single frame or a plurality of frames).

The present application contains subject matter related to thatdisclosed in Japanese Priority Patent Application JP 2009-179394 filedin the Japan Patent Office on Jul. 31, 2009, the entire content of whichis hereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus comprising:circuitry configured to decode a bit stream including filter controlinformation representing whether or not filtering is performed for anarea of a control block overlapping a processing-target slice of aplurality of slices formed in a frame, the control block being a controlunit for filtering the processing-target slice, and the area of thecontrol block is smaller than an area of the processing-target slice;and perform, according to the filter control information, filtering fora decoded image, wherein the area of the control block is specifiedbased on a control block size and a number of control blocks for anentire frame.
 2. The image processing apparatus according to claim 1,wherein the circuitry is further configured to calculate a filtercoefficient for the filtering.
 3. An image processing method comprising:decoding, with circuitry, a bit stream including filter controlinformation representing whether or not filtering is performed for anarea of a control block overlapping a processing-target slice of aplurality of slices formed in a frame, the control block being a controlunit for filtering the processing-target slice, and the area of thecontrol block is smaller than an area of the processing-target slice;and performing, with the circuitry, according to the filter controlinformation, filtering for a decoded image, wherein the area of thecontrol block is specified based on a control block size and a number ofcontrol blocks for an entire frame.
 4. The image processing methodaccording to claim 3, wherein the circuitry is further configured tocalculate a filter coefficient for the filtering.
 5. A non-transitorycomputer-readable storage medium storing thereon computer-executableinstructions which when executed by circuitry cause the circuitry toperform an image processing method, the method comprising: decoding abit stream including filter control information representing whether ornot filtering is performed for an area of a control block overlapping aprocessing-target slice of a plurality of slices formed in a frame, thecontrol block being a control unit for filtering the processing-targetslice, and the area of the control block is smaller than an area of theprocessing-target slice; and performing according to the filter controlinformation, filtering for a decoded image, wherein the area of thecontrol block is specified based on a control block size and a number ofcontrol blocks for an entire frame.
 6. The non-transitorycomputer-readable storage medium according to claim 5, wherein thecircuitry is further configured to calculate a filter coefficient forthe filtering.